[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:30 dist-test-slave-bw88 krb5kdc[5967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:30 dist-test-slave-bw88 krb5kdc[5967](info): set up 2 sockets
Aug 08 02:07:30 dist-test-slave-bw88 krb5kdc[5967](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 02:07:31.465914 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:31 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:31.601456 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:39649
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:31.721117 5983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:31.721515 5983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:31.721619 5983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:31.725639 5983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:31.725715 5983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:31.725752 5983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:31.725808 5983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:31.725847 5983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:31.731246 5983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:39649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:31.732528 5983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:31.733695 5983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:31.742877 5991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:31.743032 5983 server_base.cc:1034] running on GCE node
W20260808 02:07:31.742916 5993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:31.743227 5990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:31.743818 5983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:31.745010 5983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:31.746184 5983 hybrid_clock.cc:648] HybridClock initialized: now 1786154851746153 us; error 92 us; skew 500 ppm
Aug 08 02:07:31 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154851, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:31.749262 5983 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:31.750236 5983 webserver.cc:460] Webserver started at http://127.5.209.126:41185/ using document root <none> and password file <none>
I20260808 02:07:31.750880 5983 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:31.750989 5983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:31.751236 5983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:31.753511 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "ef05498d4869419381602fbbad0b4052"
format_stamp: "Formatted at 2026-08-08 02:07:31 on dist-test-slave-bw88"
server_key: "2aa80d4034863e554641b336979b7b08"
server_key_iv: "bc072a6cbfe4437ebb26150617ee9f3b"
server_key_version: "encryptionkey@0"
I20260808 02:07:31.754117 5983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "ef05498d4869419381602fbbad0b4052"
format_stamp: "Formatted at 2026-08-08 02:07:31 on dist-test-slave-bw88"
server_key: "2aa80d4034863e554641b336979b7b08"
server_key_iv: "bc072a6cbfe4437ebb26150617ee9f3b"
server_key_version: "encryptionkey@0"
I20260808 02:07:31.758059 5983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 02:07:31.760587 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:31.761945 5983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:07:31.762092 5983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "ef05498d4869419381602fbbad0b4052"
format_stamp: "Formatted at 2026-08-08 02:07:31 on dist-test-slave-bw88"
server_key: "2aa80d4034863e554641b336979b7b08"
server_key_iv: "bc072a6cbfe4437ebb26150617ee9f3b"
server_key_version: "encryptionkey@0"
I20260808 02:07:31.762223 5983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:31.782099 5983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:31.785120 5983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:31.785354 5983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:31.793038 5983 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:39649
I20260808 02:07:31.793069 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:39649 every 8 connection(s)
I20260808 02:07:31.794193 5983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:31.797228 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:31.799166 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 5983
I20260808 02:07:31.799342 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:31.800694 5957 external_mini_cluster.cc:1428] Setting key 0082276a1eac147f6c6b991cbdb15122
I20260808 02:07:31.803480 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052: Bootstrap starting.
I20260808 02:07:31.806349 6070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:31.807119 6070 log.cc:824] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:31.809089 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052: No bootstrap required, opened a new log
I20260808 02:07:31.813805 6070 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } }
I20260808 02:07:31.814052 6070 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:31.814139 6070 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef05498d4869419381602fbbad0b4052, State: Initialized, Role: FOLLOWER
I20260808 02:07:31.814677 6070 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } }
I20260808 02:07:31.814872 6070 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:31.814953 6070 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:31.815115 6070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:31.816412 6070 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } }
I20260808 02:07:31.816794 6070 leader_election.cc:302] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef05498d4869419381602fbbad0b4052; no voters:
I20260808 02:07:31.817206 6070 leader_election.cc:288] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:31.817251 6075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:31.817521 6075 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [term 1 LEADER]: Becoming Leader. State: Replica: ef05498d4869419381602fbbad0b4052, State: Running, Role: LEADER
I20260808 02:07:31.817884 6075 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } }
I20260808 02:07:31.818683 6070 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:31.819810 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef05498d4869419381602fbbad0b4052. Latest consensus state: current_term: 1 leader_uuid: "ef05498d4869419381602fbbad0b4052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } } }
I20260808 02:07:31.819995 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:31.820770 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef05498d4869419381602fbbad0b4052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05498d4869419381602fbbad0b4052" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39649 } } }
I20260808 02:07:31.820937 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:31.821870 6087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 02:07:31 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:31.826663 6087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:31.831763 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:31.804146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.5.209.126:39649)
0808 02:07:31.804608 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:31.804629 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:31.805740 (+ 1111us) server_negotiation.cc:407] Connection header received
0808 02:07:31.809861 (+ 4121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:31.809881 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:31.810211 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:31.810544 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:31.813244 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:31.814663 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:31.816246 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:31.816826 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:31.817042 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:31.823541 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:31.823549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:31.823567 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:31.823605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:31.826536 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:31.827238 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:31.827246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:31.827252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:31.827423 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:31.827939 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:31.827943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:31.827945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:31.828602 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:31.828931 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:31.829762 (+ 831us) server_negotiation.cc:299] Negotiation successful
0808 02:07:31.830091 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20260808 02:07:31.834605 6087 catalog_manager.cc:1347] Generated new cluster ID: ea80056c73314e3c8ede6f4ae68ccb20
I20260808 02:07:31.834676 6087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:31.843870 6087 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:31.845161 6087 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:31.867388 6087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef05498d4869419381602fbbad0b4052: Generated new TSK 0
I20260808 02:07:31.868287 6087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:31.909761 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:32.089129 6100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:32.089519 6100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:32.089604 6100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:32.093915 6100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:32.093977 6100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:32.094079 6100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:32.099620 6100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:32.100796 6100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:32.101969 6100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:32.112499 6111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.112569 6108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.112610 6109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:32.112694 6100 server_base.cc:1034] running on GCE node
I20260808 02:07:32.113227 6100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:32.113951 6100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:32.115124 6100 hybrid_clock.cc:648] HybridClock initialized: now 1786154852115106 us; error 25 us; skew 500 ppm
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:32.118780 6100 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:32.119768 6100 webserver.cc:460] Webserver started at http://127.5.209.65:41653/ using document root <none> and password file <none>
I20260808 02:07:32.120404 6100 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:32.120471 6100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:32.120730 6100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:32.123307 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "4381f40ca951435eb5ae66f181999963"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "43c18c5fabcac1d0adc9456e5cf3b05b"
server_key_iv: "b725857ed9788fccce2d3ee0f121a7bc"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.123924 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "4381f40ca951435eb5ae66f181999963"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "43c18c5fabcac1d0adc9456e5cf3b05b"
server_key_iv: "b725857ed9788fccce2d3ee0f121a7bc"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.128010 6100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 02:07:32.130780 6120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.132158 6100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:07:32.132314 6100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "4381f40ca951435eb5ae66f181999963"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "43c18c5fabcac1d0adc9456e5cf3b05b"
server_key_iv: "b725857ed9788fccce2d3ee0f121a7bc"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.132429 6100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:32.148829 6100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:32.152591 6100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:32.152859 6100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:32.153704 6100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:32.154738 6100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:32.154789 6100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.154855 6100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:32.154886 6100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.172205 6100 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:38007
I20260808 02:07:32.172394 6269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:38007 every 8 connection(s)
I20260808 02:07:32.173257 6100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:32.179260 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6100
I20260808 02:07:32.179394 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:32.179665 5957 external_mini_cluster.cc:1428] Setting key 69eba67581e0ebfa87e36f4476d99a71
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:32.189617 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.175285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:58461 (local address 127.5.209.126:39649)
0808 02:07:32.175452 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:32.175455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:32.176428 (+ 973us) server_negotiation.cc:407] Connection header received
0808 02:07:32.177938 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:32.177942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:32.178000 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:32.178110 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:32.179333 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.179995 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.181471 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.181714 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.181830 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.184993 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:32.184998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:32.185002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:32.185038 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:32.186957 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.187516 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.187523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.187528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.187641 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.188113 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.188119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.188123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.188447 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:32.188563 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:32.189138 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 02:07:32.189315 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:07:32.190440 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.175711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:39649 (local address 127.5.209.65:58461)
0808 02:07:32.176272 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:32.176316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:32.177722 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:32.178295 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:32.178302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:32.178674 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:32.179180 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.179195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.180130 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.180134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.181283 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.181293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.181877 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.181882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.182068 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.182845 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:32.182864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:32.184795 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:32.187093 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.187098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.187111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.187401 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.187785 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.187788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.187789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.187945 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.188570 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:32.188576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:32.188847 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 02:07:32.189039 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 02:07:32.189325 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":208,"threads_started":1}
I20260808 02:07:32.191875 6270 heartbeater.cc:347] Connected to a master server at 127.5.209.126:39649
I20260808 02:07:32.192176 6270 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:32.192757 6270 heartbeater.cc:511] Master 127.5.209.126:39649 requested a full tablet report, sending...
I20260808 02:07:32.194506 6023 ts_manager.cc:194] Registered new tserver with Master: 4381f40ca951435eb5ae66f181999963 (127.5.209.65:38007)
I20260808 02:07:32.195837 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:58461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:32.217236 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:32.388666 6275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:32.388917 6275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:32.388986 6275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:32.393108 6275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:32.393159 6275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:32.393265 6275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:32.398708 6275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:32.399749 6275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:32.400755 6275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:32.411090 6284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.411181 6280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.411070 6281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:32.411340 6275 server_base.cc:1034] running on GCE node
I20260808 02:07:32.411810 6275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:32.412482 6275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:32.413676 6275 hybrid_clock.cc:648] HybridClock initialized: now 1786154852413650 us; error 46 us; skew 500 ppm
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:32.417203 6275 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:32.418184 6275 webserver.cc:460] Webserver started at http://127.5.209.66:41983/ using document root <none> and password file <none>
I20260808 02:07:32.418785 6275 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:32.418882 6275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:32.419183 6275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:32.421752 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "c5e69a47e7374dfebc67c57eda3b94e4"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "5cee92ac6729949532d56d4e3cf0c846"
server_key_iv: "f2e2fa8752fe8bfd412112e1a5f693bb"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.422349 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "c5e69a47e7374dfebc67c57eda3b94e4"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "5cee92ac6729949532d56d4e3cf0c846"
server_key_iv: "f2e2fa8752fe8bfd412112e1a5f693bb"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.426519 6275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:07:32.429203 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.430564 6275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:07:32.430672 6275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "c5e69a47e7374dfebc67c57eda3b94e4"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "5cee92ac6729949532d56d4e3cf0c846"
server_key_iv: "f2e2fa8752fe8bfd412112e1a5f693bb"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.430757 6275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:32.446877 6275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:32.450518 6275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:32.450742 6275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:32.451463 6275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:32.452457 6275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:32.452505 6275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.452553 6275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:32.452606 6275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.473726 6275 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:46359
I20260808 02:07:32.473887 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:46359 every 8 connection(s)
I20260808 02:07:32.474748 6275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:32.475983 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6275
I20260808 02:07:32.476078 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:32.476379 5957 external_mini_cluster.cc:1428] Setting key 76c4b8864d03bebf18ff476416dae26c
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:07:32.491518 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.476985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:60127 (local address 127.5.209.126:39649)
0808 02:07:32.477140 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:32.477145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:32.478444 (+ 1299us) server_negotiation.cc:407] Connection header received
0808 02:07:32.480156 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:32.480160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:32.480215 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:32.480295 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:32.481990 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.482536 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.483854 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.484098 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.484216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.487408 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:32.487412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:32.487415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:32.487450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:32.489134 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.489710 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.489714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.489716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.489846 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.490314 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.490317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.490319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.490516 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:32.490636 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:32.491144 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 02:07:32.491313 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:07:32.492432 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.477479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:39649 (local address 127.5.209.66:60127)
0808 02:07:32.478266 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:32.478320 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:32.479923 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:32.480472 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:32.480479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:32.481107 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:32.481836 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.481853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.482670 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.482673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.483719 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.483729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.484261 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.484265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.484471 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.485171 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:32.485188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:32.487241 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:32.489253 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.489259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.489274 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.489600 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.489951 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.489955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.489957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.490167 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.490636 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:32.490643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:32.490887 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 02:07:32.491081 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 02:07:32.491410 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":301,"threads_started":1}
I20260808 02:07:32.493706 6437 heartbeater.cc:347] Connected to a master server at 127.5.209.126:39649
I20260808 02:07:32.494033 6437 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:32.494774 6437 heartbeater.cc:511] Master 127.5.209.126:39649 requested a full tablet report, sending...
I20260808 02:07:32.495889 6023 ts_manager.cc:194] Registered new tserver with Master: c5e69a47e7374dfebc67c57eda3b94e4 (127.5.209.66:46359)
I20260808 02:07:32.496472 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:60127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:32.511966 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:32.687373 6442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:32.687654 6442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:32.687700 6442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:32.691861 6442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:32.691916 6442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:32.691975 6442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:32.697738 6442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:39649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:32.698760 6442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:32.699772 6442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:32.711508 6452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.711589 6450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:32.711612 6449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:32.711751 6442 server_base.cc:1034] running on GCE node
I20260808 02:07:32.712294 6442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:32.712984 6442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:32.714174 6442 hybrid_clock.cc:648] HybridClock initialized: now 1786154852714160 us; error 25 us; skew 500 ppm
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:32.717618 6442 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:32.718622 6442 webserver.cc:460] Webserver started at http://127.5.209.67:34973/ using document root <none> and password file <none>
I20260808 02:07:32.719290 6442 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:32.719394 6442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:32.719658 6442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:32.722366 6442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "c17c0eabd33c4816b6dd0b2cfd2eeda5"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "840a28c8782d696ea7f35fba207270e0"
server_key_iv: "d258c11959587d8869615a81bd8a2e83"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.723114 6442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "c17c0eabd33c4816b6dd0b2cfd2eeda5"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "840a28c8782d696ea7f35fba207270e0"
server_key_iv: "d258c11959587d8869615a81bd8a2e83"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.727612 6442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:32.730937 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.732285 6442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:07:32.732434 6442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "c17c0eabd33c4816b6dd0b2cfd2eeda5"
format_stamp: "Formatted at 2026-08-08 02:07:32 on dist-test-slave-bw88"
server_key: "840a28c8782d696ea7f35fba207270e0"
server_key_iv: "d258c11959587d8869615a81bd8a2e83"
server_key_version: "encryptionkey@0"
I20260808 02:07:32.732559 6442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:32.748291 6442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:32.751677 6442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:32.751958 6442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:32.752820 6442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:32.753933 6442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:32.754014 6442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.754091 6442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:32.754141 6442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:32.777062 6442 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:43199
I20260808 02:07:32.777208 6607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:43199 every 8 connection(s)
I20260808 02:07:32.778218 6442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:32.781232 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6442
I20260808 02:07:32.781394 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:32.781822 5957 external_mini_cluster.cc:1428] Setting key ae2002e2520743448dd975900a585aca
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:32.795339 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.780624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:52849 (local address 127.5.209.126:39649)
0808 02:07:32.780862 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:32.780866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:32.782041 (+ 1175us) server_negotiation.cc:407] Connection header received
0808 02:07:32.784200 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:32.784205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:32.784266 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:32.784361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:32.786065 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.786649 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.788197 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.788415 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.788548 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.791473 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:32.791476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:32.791478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:32.791502 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:32.793158 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.793716 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.793720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.793722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.793808 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.794156 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.794159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.794161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.794366 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:32.794483 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:32.795066 (+ 583us) server_negotiation.cc:299] Negotiation successful
0808 02:07:32.795185 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260808 02:07:32.796425 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.781018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:39649 (local address 127.5.209.67:52849)
0808 02:07:32.781836 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:32.781902 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:32.783985 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:32.784528 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:32.784535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:32.785146 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:32.785894 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.785911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.786768 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.786770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.788062 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:32.788073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.788608 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.788612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:32.788845 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.789591 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:32.789608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:32.791319 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:32.793279 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.793284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.793296 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.793605 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.793900 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:32.793902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:32.793904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:32.794057 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:32.794504 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:32.794510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:32.794763 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 02:07:32.794991 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 02:07:32.795348 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":288,"threads_started":1}
I20260808 02:07:32.797680 6610 heartbeater.cc:347] Connected to a master server at 127.5.209.126:39649
I20260808 02:07:32.797927 6610 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:32.798467 6610 heartbeater.cc:511] Master 127.5.209.126:39649 requested a full tablet report, sending...
I20260808 02:07:32.799552 6023 ts_manager.cc:194] Registered new tserver with Master: c17c0eabd33c4816b6dd0b2cfd2eeda5 (127.5.209.67:43199)
I20260808 02:07:32.800149 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:52849
I20260808 02:07:32.808559 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:32.839676 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.828348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.5.209.65:38007)
0808 02:07:32.828739 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:32.828745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:32.828800 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:07:32.828959 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:32.828965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:32.829183 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:32.829304 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:32.830244 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.831190 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.832173 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.832642 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.832733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.835665 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:32.835674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:32.835688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:32.835713 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:32.837594 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.838144 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.838149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.838151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.838237 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.838556 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.838559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.838561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.838831 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:32.839082 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:32.839249 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 02:07:32.839422 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":176,"threads_started":1}
W20260808 02:07:32.840909 6155 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[5967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:32.872082 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:32.862501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.5.209.65:38007)
0808 02:07:32.862681 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:32.862687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:32.862799 (+ 112us) server_negotiation.cc:407] Connection header received
0808 02:07:32.863000 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:32.863008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:32.863057 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:32.863159 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:32.864006 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.864571 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.865667 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:32.865920 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:32.866017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:32.868806 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:32.868809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:32.868811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:32.868837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:32.870412 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.870884 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.870888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.870889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.870987 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:32.871238 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:32.871240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:32.871241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:32.871408 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:32.871658 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:32.871747 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:07:32.871853 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:07:32.873250 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6100
I20260808 02:07:32.886897 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6275
I20260808 02:07:32.897578 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6442
I20260808 02:07:32.911099 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 5983
2026-08-08T02:07:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2787 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[6627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[6627](info): set up 2 sockets
Aug 08 02:07:32 dist-test-slave-bw88 krb5kdc[6627](info): commencing operation
krb5kdc: starting...
W20260808 02:07:34.111320 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:34.244086 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34727
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:34.372912 6641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:34.373210 6641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:34.373323 6641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:34.377476 6641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:34.377552 6641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:34.377573 6641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:34.377691 6641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:34.377729 6641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:34.384521 6641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:34.385921 6641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:34.387217 6641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:34.397215 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:34.397451 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:34.397207 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:34.397572 6641 server_base.cc:1034] running on GCE node
I20260808 02:07:34.398247 6641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:34.399740 6641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:34.400975 6641 hybrid_clock.cc:648] HybridClock initialized: now 1786154854400945 us; error 40 us; skew 500 ppm
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:34.404415 6641 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:34.405612 6641 webserver.cc:460] Webserver started at http://127.5.209.126:41899/ using document root <none> and password file <none>
I20260808 02:07:34.406436 6641 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:34.406539 6641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:34.406821 6641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:34.409894 6641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "28f84fbaf7374614a5a7e631074427f2"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "8c9c728156c361bfd5d517b0e02067b7"
server_key_iv: "987e54cba3f90e79ca4ec248e32ea212"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.410691 6641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "28f84fbaf7374614a5a7e631074427f2"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "8c9c728156c361bfd5d517b0e02067b7"
server_key_iv: "987e54cba3f90e79ca4ec248e32ea212"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.416018 6641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:07:34.419193 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.421060 6641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:07:34.421234 6641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "28f84fbaf7374614a5a7e631074427f2"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "8c9c728156c361bfd5d517b0e02067b7"
server_key_iv: "987e54cba3f90e79ca4ec248e32ea212"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.421478 6641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:34.437850 6641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:34.441269 6641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:34.441556 6641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:34.450029 6641 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34727
I20260808 02:07:34.450070 6725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34727 every 8 connection(s)
I20260808 02:07:34.451152 6641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:34.454314 6726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:34.460110 6726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2: Bootstrap starting.
I20260808 02:07:34.460942 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6641
I20260808 02:07:34.461124 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:34.461472 5957 external_mini_cluster.cc:1428] Setting key a6b658ab7ce94b95ffff3d9aca0a4d9d
I20260808 02:07:34.463027 6726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:34.463821 6726 log.cc:824] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:34.466171 6726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2: No bootstrap required, opened a new log
I20260808 02:07:34.470553 6726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } }
I20260808 02:07:34.470855 6726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:34.470937 6726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f84fbaf7374614a5a7e631074427f2, State: Initialized, Role: FOLLOWER
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:34.471477 6726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } }
I20260808 02:07:34.471632 6726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:34.471701 6726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:34.471805 6726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:34.472692 6726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } }
I20260808 02:07:34.473057 6726 leader_election.cc:302] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f84fbaf7374614a5a7e631074427f2; no voters:
I20260808 02:07:34.473430 6726 leader_election.cc:288] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:34.473512 6732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:34.473702 6732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [term 1 LEADER]: Becoming Leader. State: Replica: 28f84fbaf7374614a5a7e631074427f2, State: Running, Role: LEADER
I20260808 02:07:34.473994 6732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } }
I20260808 02:07:34.474727 6726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:34.476130 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f84fbaf7374614a5a7e631074427f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } } }
I20260808 02:07:34.476727 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:34.476503 6733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f84fbaf7374614a5a7e631074427f2. Latest consensus state: current_term: 1 leader_uuid: "28f84fbaf7374614a5a7e631074427f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f84fbaf7374614a5a7e631074427f2" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34727 } } }
I20260808 02:07:34.477648 6733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:34.478015 6747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:34.479125 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:34.462816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.5.209.126:34727)
0808 02:07:34.463474 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:34.463493 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:34.463534 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:07:34.464291 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:34.464311 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:34.464663 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:34.465000 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:34.465961 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.467154 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.468374 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.468896 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.469077 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:34.472369 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:34.472376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:34.472391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:34.472466 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:34.474858 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.475519 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.475529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.475538 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.475706 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.476089 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.476095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.476097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.476629 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:34.477531 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:34.477738 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 02:07:34.478096 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":257,"threads_started":1}
I20260808 02:07:34.482990 6747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:34.488961 6747 catalog_manager.cc:1347] Generated new cluster ID: ea47439bff22445d94be2b114feb13de
I20260808 02:07:34.489090 6747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:34.496459 6747 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:34.497465 6747 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:34.505499 6747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f84fbaf7374614a5a7e631074427f2: Generated new TSK 0
I20260808 02:07:34.506078 6747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:34.544814 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:34.669680 6757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:34.670003 6757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:34.670094 6757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:34.674093 6757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:34.674185 6757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:34.674279 6757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:34.679900 6757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:34.680994 6757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:34.682111 6757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:34.692632 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:34.692664 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:34.692664 6771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:34.692721 6757 server_base.cc:1034] running on GCE node
I20260808 02:07:34.693245 6757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:34.693914 6757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:34.695092 6757 hybrid_clock.cc:648] HybridClock initialized: now 1786154854695078 us; error 27 us; skew 500 ppm
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:34.698212 6757 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:34.699329 6757 webserver.cc:460] Webserver started at http://127.5.209.65:45733/ using document root <none> and password file <none>
I20260808 02:07:34.699942 6757 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:34.700016 6757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:34.700251 6757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:34.702522 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "58837d5abd9f48d7b71f81a02e2a390f"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "c3b20d12dab05830717bd634cc795f38"
server_key_iv: "b19cfc71621b7cce8ae4324d62c77ddb"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.703154 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "58837d5abd9f48d7b71f81a02e2a390f"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "c3b20d12dab05830717bd634cc795f38"
server_key_iv: "b19cfc71621b7cce8ae4324d62c77ddb"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.707140 6757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 02:07:34.709734 6779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.711038 6757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:07:34.711175 6757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "58837d5abd9f48d7b71f81a02e2a390f"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "c3b20d12dab05830717bd634cc795f38"
server_key_iv: "b19cfc71621b7cce8ae4324d62c77ddb"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.711297 6757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:34.732761 6757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:34.735993 6757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:34.736241 6757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:34.736925 6757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:34.737983 6757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:34.738065 6757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.738147 6757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:34.738194 6757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.748353 6757 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:38131
I20260808 02:07:34.748373 6918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:38131 every 8 connection(s)
I20260808 02:07:34.749390 6757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:34.752110 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6757
I20260808 02:07:34.752247 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:34.752568 5957 external_mini_cluster.cc:1428] Setting key e9982738f09a721a5b51fc1ee6537512
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:34.764710 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:34.751320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:56879 (local address 127.5.209.126:34727)
0808 02:07:34.751478 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:34.751482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:34.752390 (+ 908us) server_negotiation.cc:407] Connection header received
0808 02:07:34.754229 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:34.754232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:34.754285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:34.754375 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:34.755684 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.756513 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.757784 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.758029 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.758138 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:34.761076 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:34.761079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:34.761081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:34.761105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:34.762577 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.763106 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.763109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.763111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.763196 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.763604 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.763609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.763610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.763793 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:34.763905 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:34.764474 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 02:07:34.764581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:07:34.765825 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:34.751684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34727 (local address 127.5.209.65:56879)
0808 02:07:34.752224 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:34.752277 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:34.753970 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:34.754603 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:34.754611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:34.754987 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:34.755513 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:34.755528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.756651 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.756656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:34.757653 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:34.757663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.758194 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.758198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:34.758371 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:34.759095 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:34.759112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:34.760887 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:34.762700 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:34.762706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:34.762721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:34.762987 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:34.763295 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:34.763299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:34.763301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:34.763496 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:34.763905 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:34.763913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:34.764178 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 02:07:34.764414 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 02:07:34.764717 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:07:34.767344 6919 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34727
I20260808 02:07:34.767668 6919 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:34.768302 6919 heartbeater.cc:511] Master 127.5.209.126:34727 requested a full tablet report, sending...
I20260808 02:07:34.770125 6678 ts_manager.cc:194] Registered new tserver with Master: 58837d5abd9f48d7b71f81a02e2a390f (127.5.209.65:38131)
I20260808 02:07:34.771288 6678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:56879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:34.783995 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:34.901968 6929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:34.902204 6929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:34.902245 6929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:34.906057 6929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:34.906103 6929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:34.906158 6929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:34.911229 6929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:34.912164 6929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:34.913111 6929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:34.923062 6934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:34.923082 6938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:34.923169 6929 server_base.cc:1034] running on GCE node
W20260808 02:07:34.923070 6935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:34.923679 6929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:34.924381 6929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:34.925550 6929 hybrid_clock.cc:648] HybridClock initialized: now 1786154854925530 us; error 31 us; skew 500 ppm
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:34.928412 6929 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:34.929391 6929 webserver.cc:460] Webserver started at http://127.5.209.66:43881/ using document root <none> and password file <none>
I20260808 02:07:34.930052 6929 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:34.930148 6929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:34.930366 6929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:34.932440 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "f8f58442c82a40c28157b05437a07906"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "90118d02b864eaba97db813061be6887"
server_key_iv: "f61c181cb2dce7c09818dd655e2e7b03"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.933022 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "f8f58442c82a40c28157b05437a07906"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "90118d02b864eaba97db813061be6887"
server_key_iv: "f61c181cb2dce7c09818dd655e2e7b03"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.937089 6929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 02:07:34.939998 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.941425 6929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:34.941602 6929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "f8f58442c82a40c28157b05437a07906"
format_stamp: "Formatted at 2026-08-08 02:07:34 on dist-test-slave-bw88"
server_key: "90118d02b864eaba97db813061be6887"
server_key_iv: "f61c181cb2dce7c09818dd655e2e7b03"
server_key_version: "encryptionkey@0"
I20260808 02:07:34.941727 6929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:34.957706 6929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:34.960916 6929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:34.961165 6929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:34.961848 6929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:34.962747 6929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:34.962822 6929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.962898 6929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:34.962949 6929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:34.972410 6929 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:37267
I20260808 02:07:34.972434 7087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:37267 every 8 connection(s)
I20260808 02:07:34.973369 6929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:34.980952 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 6929
I20260808 02:07:34.981055 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:34.981547 5957 external_mini_cluster.cc:1428] Setting key ba3ba728924ec090bdf1ab1a4b9442ad
Aug 08 02:07:34 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:34.989039 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:34.975410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:50575 (local address 127.5.209.126:34727)
0808 02:07:34.975592 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:34.975598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:34.976550 (+ 952us) server_negotiation.cc:407] Connection header received
0808 02:07:34.978154 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:34.978157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:34.978208 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:34.978287 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:34.979542 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.980068 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.981359 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.981689 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.981807 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:34.984834 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:34.984841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:34.984847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:34.984885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:34.986535 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.987217 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.987220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.987221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.987308 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:34.987748 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:34.987751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:34.987752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:34.987938 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:34.988025 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:34.988787 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 02:07:34.988909 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:07:34.990322 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:34.975879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34727 (local address 127.5.209.66:50575)
0808 02:07:34.976391 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:34.976435 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:34.977898 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:34.978458 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:34.978465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:34.978826 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:34.979343 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:34.979357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.980192 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.980195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:34.981213 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:34.981224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:34.981842 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:34.981845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:34.982044 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:34.982778 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:34.982793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:34.984659 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:34.986665 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:34.986671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:34.986686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:34.987100 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:34.987416 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:34.987422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:34.987426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:34.987637 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:34.988059 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:34.988068 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:34.988451 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 02:07:34.988718 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 02:07:34.989107 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260808 02:07:34.991611 7088 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34727
I20260808 02:07:34.991899 7088 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:34.992565 7088 heartbeater.cc:511] Master 127.5.209.126:34727 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:07:34.993919 6678 ts_manager.cc:194] Registered new tserver with Master: f8f58442c82a40c28157b05437a07906 (127.5.209.66:37267)
I20260808 02:07:34.994546 6678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:50575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:35.012862 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:35.135345 7094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:35.135667 7094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:35.135747 7094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:35.139945 7094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:35.140034 7094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:35.140132 7094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:35.145617 7094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:35.146874 7094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:35.148005 7094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:35.158878 7100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:35.158854 7094 server_base.cc:1034] running on GCE node
W20260808 02:07:35.158838 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:35.158847 7101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:35.159735 7094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:35.160476 7094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:35.161708 7094 hybrid_clock.cc:648] HybridClock initialized: now 1786154855161670 us; error 50 us; skew 500 ppm
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:35.165047 7094 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:35.166028 7094 webserver.cc:460] Webserver started at http://127.5.209.67:46141/ using document root <none> and password file <none>
I20260808 02:07:35.166595 7094 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:35.166657 7094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:35.166833 7094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:35.168933 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "5887454467004c829ff3bc5eb6aa3a4c"
format_stamp: "Formatted at 2026-08-08 02:07:35 on dist-test-slave-bw88"
server_key: "425e8ecff08eda985c3358eebc22b10a"
server_key_iv: "01c7520cab9c46f6f1c65d396cb712af"
server_key_version: "encryptionkey@0"
I20260808 02:07:35.169562 7094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "5887454467004c829ff3bc5eb6aa3a4c"
format_stamp: "Formatted at 2026-08-08 02:07:35 on dist-test-slave-bw88"
server_key: "425e8ecff08eda985c3358eebc22b10a"
server_key_iv: "01c7520cab9c46f6f1c65d396cb712af"
server_key_version: "encryptionkey@0"
I20260808 02:07:35.173466 7094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:35.176376 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:35.177842 7094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:35.177995 7094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "5887454467004c829ff3bc5eb6aa3a4c"
format_stamp: "Formatted at 2026-08-08 02:07:35 on dist-test-slave-bw88"
server_key: "425e8ecff08eda985c3358eebc22b10a"
server_key_iv: "01c7520cab9c46f6f1c65d396cb712af"
server_key_version: "encryptionkey@0"
I20260808 02:07:35.178107 7094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:35.195423 7094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:35.198949 7094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:35.199172 7094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:35.199869 7094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:35.200802 7094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:35.200863 7094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:35.200930 7094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:35.200963 7094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:35.210801 7094 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:35767
I20260808 02:07:35.210912 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:35767 every 8 connection(s)
I20260808 02:07:35.211792 7094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:35.220381 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 7094
I20260808 02:07:35.220520 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:35.220851 5957 external_mini_cluster.cc:1428] Setting key 6874a4e5daa4f0b2761972c496089b20
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:35.236892 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.213851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:35753 (local address 127.5.209.126:34727)
0808 02:07:35.214042 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.214047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.214918 (+ 871us) server_negotiation.cc:407] Connection header received
0808 02:07:35.216331 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.216334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.216396 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.216488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:35.217886 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.218432 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.219641 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.219857 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.219944 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.233005 (+ 13061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:35.233009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:35.233011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:35.233038 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:35.234754 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.235462 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.235465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.235467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.235564 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.235862 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.235864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.235865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.236072 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:35.236151 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.236663 (+ 512us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.236768 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:07:35.237851 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.214248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34727 (local address 127.5.209.67:35753)
0808 02:07:35.214757 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.214802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.216098 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.216725 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.216733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.217168 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:35.217720 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.217743 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.218574 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.218577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.219519 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.219530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.219997 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.219999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.220138 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.230943 (+ 10805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:35.230972 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:35.232784 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:35.234895 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:35.234903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:35.234921 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:35.235346 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:35.235635 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:35.235637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:35.235638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:35.235782 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:35.236168 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:35.236174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:35.236434 (+ 260us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.236614 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.236903 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
I20260808 02:07:35.239215 7269 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34727
I20260808 02:07:35.239451 7269 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:35.240026 7269 heartbeater.cc:511] Master 127.5.209.126:34727 requested a full tablet report, sending...
I20260808 02:07:35.241065 6678 ts_manager.cc:194] Registered new tserver with Master: 5887454467004c829ff3bc5eb6aa3a4c (127.5.209.67:35767)
I20260808 02:07:35.241775 6678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:35753
I20260808 02:07:35.247283 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:07:35.259917 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.251718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.5.209.126:34727)
0808 02:07:35.251904 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.251908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.252245 (+ 337us) server_negotiation.cc:407] Connection header received
0808 02:07:35.252390 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.252392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.252434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.252508 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:35.253546 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.254075 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.255002 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.255220 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.255314 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.256826 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:35.256829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:35.256831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:35.256855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:35.258410 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.258814 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.258817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.258818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.258893 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.259131 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.259133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.259134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.259347 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:35.259450 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.259703 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.259797 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:07:35.264995 6678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:35.268309 6678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:35.283596 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.277177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:38131 (local address 127.0.0.1:34248)
0808 02:07:35.278188 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.278202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.278320 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.279318 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.279321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.279338 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:35.279422 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.279428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.280562 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.280566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.281765 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.281777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.282933 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.282936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.283140 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.283199 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.283322 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.283446 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260808 02:07:35.283939 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.278016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.5.209.65:38131)
0808 02:07:35.278481 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.278489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.278514 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:07:35.278595 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.278600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.278790 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.278951 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:35.279550 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.280439 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.281913 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.282771 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.282940 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.283138 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.283363 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.283509 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":75,"threads_started":1}
I20260808 02:07:35.284372 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.277060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:35767 (local address 127.0.0.1:50708)
0808 02:07:35.277454 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.277495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.277601 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.278577 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.278583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.278607 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:35.278825 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.278836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.281359 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.281364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.282522 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.282535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.283898 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.283903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.284064 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.284099 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.284159 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.284240 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260808 02:07:35.284711 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.278334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.5.209.66:37267)
0808 02:07:35.279241 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.279296 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.279319 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:07:35.279393 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.279399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.279581 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.279789 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:35.280090 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.281536 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.283257 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.284144 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.284258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.284400 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.284468 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.284566 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":150,"threads_started":1}
I20260808 02:07:35.284727 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.277894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:37267 (local address 127.0.0.1:36158)
0808 02:07:35.279033 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.279047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.279132 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.279761 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.279764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.279784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:35.279869 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.279875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.281818 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.281821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.283121 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.283132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.284256 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.284260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.284345 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.284367 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.284515 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.284602 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":101,"threads_started":1}
I20260808 02:07:35.285251 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.277475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.5.209.67:35767)
0808 02:07:35.277932 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.277983 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.278008 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:07:35.278111 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.278118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.278357 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.278590 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:35.279573 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.281093 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.282659 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.283735 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.284660 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.284806 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.284863 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.284983 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":115,"threads_started":1}
I20260808 02:07:35.287364 6839 tablet_service.cc:1467] Processing CreateTablet for tablet 64a516afde4a40a98f93a6e7913f4d0d (DEFAULT_TABLE table=test-table [id=a6392fce97a1443793a842db9324b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:35.287487 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 64a516afde4a40a98f93a6e7913f4d0d (DEFAULT_TABLE table=test-table [id=a6392fce97a1443793a842db9324b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:35.288709 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a516afde4a40a98f93a6e7913f4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:35.288709 6839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a516afde4a40a98f93a6e7913f4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:35.291572 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 64a516afde4a40a98f93a6e7913f4d0d (DEFAULT_TABLE table=test-table [id=a6392fce97a1443793a842db9324b8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:35.292606 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a516afde4a40a98f93a6e7913f4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:35.298568 7287 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Bootstrap starting.
I20260808 02:07:35.300097 7289 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Bootstrap starting.
I20260808 02:07:35.301110 7290 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Bootstrap starting.
I20260808 02:07:35.302415 7287 tablet_bootstrap.cc:654] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:35.303748 7287 log.cc:824] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:35.304488 7289 tablet_bootstrap.cc:654] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:35.305622 7290 tablet_bootstrap.cc:654] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:35.305953 7287 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: No bootstrap required, opened a new log
I20260808 02:07:35.306198 7287 ts_tablet_manager.cc:1397] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:07:35.306357 7289 log.cc:824] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:35.307238 7290 log.cc:824] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:35.308553 7289 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: No bootstrap required, opened a new log
I20260808 02:07:35.308745 7289 ts_tablet_manager.cc:1397] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260808 02:07:35.309885 7290 tablet_bootstrap.cc:492] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: No bootstrap required, opened a new log
I20260808 02:07:35.310077 7290 ts_tablet_manager.cc:1397] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:07:35.312348 7287 raft_consensus.cc:348] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.312594 7287 raft_consensus.cc:374] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:35.312670 7287 raft_consensus.cc:729] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58837d5abd9f48d7b71f81a02e2a390f, State: Initialized, Role: FOLLOWER
I20260808 02:07:35.313153 7287 consensus_queue.cc:260] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.314082 6919 heartbeater.cc:503] Master 127.5.209.126:34727 was elected leader, sending a full tablet report...
I20260808 02:07:35.314280 7287 ts_tablet_manager.cc:1428] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260808 02:07:35.315143 7289 raft_consensus.cc:348] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.315662 7289 raft_consensus.cc:374] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:35.315771 7289 raft_consensus.cc:729] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f58442c82a40c28157b05437a07906, State: Initialized, Role: FOLLOWER
I20260808 02:07:35.316251 7289 consensus_queue.cc:260] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.316793 7290 raft_consensus.cc:348] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.317099 7290 raft_consensus.cc:374] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:35.317165 7290 raft_consensus.cc:729] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5887454467004c829ff3bc5eb6aa3a4c, State: Initialized, Role: FOLLOWER
I20260808 02:07:35.317559 7289 ts_tablet_manager.cc:1428] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260808 02:07:35.317723 7290 consensus_queue.cc:260] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.317886 7088 heartbeater.cc:503] Master 127.5.209.126:34727 was elected leader, sending a full tablet report...
I20260808 02:07:35.318620 7269 heartbeater.cc:503] Master 127.5.209.126:34727 was elected leader, sending a full tablet report...
I20260808 02:07:35.318840 7290 ts_tablet_manager.cc:1428] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260808 02:07:35.462918 7270 tablet.cc:2369] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:07:35.474761 7089 tablet.cc:2369] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:07:35.475615 7296 raft_consensus.cc:482] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:35.476037 7296 raft_consensus.cc:504] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.477445 7296 leader_election.cc:288] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5887454467004c829ff3bc5eb6aa3a4c (127.5.209.67:35767), f8f58442c82a40c28157b05437a07906 (127.5.209.66:37267)
I20260808 02:07:35.482513 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.477877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:42229 (local address 127.5.209.67:35767)
0808 02:07:35.478092 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.478111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.478445 (+ 334us) server_negotiation.cc:407] Connection header received
0808 02:07:35.478682 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.478686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.478750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.478840 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:35.479372 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.479892 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.481208 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.481822 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.481957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.482020 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.482298 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.482377 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:07:35.482599 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.477846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:35767 (local address 127.5.209.65:42229)
0808 02:07:35.478275 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.478291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.478549 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.478855 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.478858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.478992 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:35.479231 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.479237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.480024 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.480029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.481055 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.481065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.481949 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.481953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.482149 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.482172 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.482276 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.482401 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260808 02:07:35.482785 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.478449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:42549 (local address 127.5.209.66:37267)
0808 02:07:35.478639 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.478643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.478847 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:07:35.479086 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.479097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.479142 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.479226 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:35.479721 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.480214 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.481708 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.482150 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.482276 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.482368 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.482609 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.482678 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 02:07:35.482918 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.478261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:37267 (local address 127.5.209.65:42549)
0808 02:07:35.478719 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:35.478730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:35.478966 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:35.479242 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:35.479245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:35.479261 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:35.479596 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.479604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.480332 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.480336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.481571 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:35.481582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.482274 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.482277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:35.482467 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.482494 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:07:35.482589 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:07:35.482659 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":164,"threads_started":1}
I20260808 02:07:35.483657 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a516afde4a40a98f93a6e7913f4d0d" candidate_uuid: "58837d5abd9f48d7b71f81a02e2a390f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5887454467004c829ff3bc5eb6aa3a4c" is_pre_election: true
I20260808 02:07:35.483685 7027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a516afde4a40a98f93a6e7913f4d0d" candidate_uuid: "58837d5abd9f48d7b71f81a02e2a390f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f58442c82a40c28157b05437a07906" is_pre_election: true
I20260808 02:07:35.484030 7027 raft_consensus.cc:2454] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58837d5abd9f48d7b71f81a02e2a390f in term 0.
I20260808 02:07:35.484027 7204 raft_consensus.cc:2454] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58837d5abd9f48d7b71f81a02e2a390f in term 0.
I20260808 02:07:35.484575 6785 leader_election.cc:302] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58837d5abd9f48d7b71f81a02e2a390f, f8f58442c82a40c28157b05437a07906; no voters:
I20260808 02:07:35.484932 7296 raft_consensus.cc:2781] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:35.485025 7296 raft_consensus.cc:482] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:35.485085 7296 raft_consensus.cc:3037] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:35.486550 7296 raft_consensus.cc:504] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.486927 7296 leader_election.cc:288] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [CANDIDATE]: Term 1 election: Requested vote from peers 5887454467004c829ff3bc5eb6aa3a4c (127.5.209.67:35767), f8f58442c82a40c28157b05437a07906 (127.5.209.66:37267)
I20260808 02:07:35.487416 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a516afde4a40a98f93a6e7913f4d0d" candidate_uuid: "58837d5abd9f48d7b71f81a02e2a390f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5887454467004c829ff3bc5eb6aa3a4c"
I20260808 02:07:35.487563 7204 raft_consensus.cc:3037] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:35.487689 7027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a516afde4a40a98f93a6e7913f4d0d" candidate_uuid: "58837d5abd9f48d7b71f81a02e2a390f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f58442c82a40c28157b05437a07906"
I20260808 02:07:35.487828 7027 raft_consensus.cc:3037] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:35.489428 7204 raft_consensus.cc:2454] T 64a516afde4a40a98f93a6e7913f4d0d P 5887454467004c829ff3bc5eb6aa3a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58837d5abd9f48d7b71f81a02e2a390f in term 1.
I20260808 02:07:35.489787 6785 leader_election.cc:302] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58837d5abd9f48d7b71f81a02e2a390f, 5887454467004c829ff3bc5eb6aa3a4c; no voters:
I20260808 02:07:35.489835 7027 raft_consensus.cc:2454] T 64a516afde4a40a98f93a6e7913f4d0d P f8f58442c82a40c28157b05437a07906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58837d5abd9f48d7b71f81a02e2a390f in term 1.
I20260808 02:07:35.490022 7296 raft_consensus.cc:2781] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:35.490350 7296 raft_consensus.cc:686] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [term 1 LEADER]: Becoming Leader. State: Replica: 58837d5abd9f48d7b71f81a02e2a390f, State: Running, Role: LEADER
I20260808 02:07:35.490741 7296 consensus_queue.cc:237] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } }
I20260808 02:07:35.494243 6678 catalog_manager.cc:5495] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f reported cstate change: term changed from 0 to 1, leader changed from <none> to 58837d5abd9f48d7b71f81a02e2a390f (127.5.209.65). New cstate: current_term: 1 leader_uuid: "58837d5abd9f48d7b71f81a02e2a390f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5887454467004c829ff3bc5eb6aa3a4c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58837d5abd9f48d7b71f81a02e2a390f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8f58442c82a40c28157b05437a07906" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20260808 02:07:35.500893 6922 tablet.cc:2369] T 64a516afde4a40a98f93a6e7913f4d0d P 58837d5abd9f48d7b71f81a02e2a390f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:35.522835 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.512087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.5.209.65:38131)
0808 02:07:35.512305 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.512310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.512611 (+ 301us) server_negotiation.cc:407] Connection header received
0808 02:07:35.512719 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.512721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.512766 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.512836 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:35.513795 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.514278 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.515458 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.515781 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.515881 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.518655 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:35.518667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:35.518682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:35.518705 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:35.520765 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.521284 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.521290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.521291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.521564 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.521837 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.521840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.521841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.522096 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:35.522263 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.522515 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.522664 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:35.555915 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.546270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.5.209.65:38131)
0808 02:07:35.546437 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.546442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.546576 (+ 134us) server_negotiation.cc:407] Connection header received
0808 02:07:35.546765 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.546769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.546825 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.546916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:35.547620 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.548185 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.549120 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.549428 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.549557 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.552223 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:35.552227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:35.552229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:35.552255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:35.554042 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.554534 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.554538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.554540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.554630 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.554916 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.554919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.554921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.555120 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:35.555288 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.555610 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.555743 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[6627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:35.585688 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:35.576335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.5.209.65:38131)
0808 02:07:35.576511 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:35.576515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:35.576683 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:07:35.576809 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:35.576812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:35.576859 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:35.576933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:35.577754 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.578266 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.579152 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:35.579369 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:35.579453 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:35.582425 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:35.582428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:35.582431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:35.582455 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:35.584019 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.584420 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.584422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.584424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.584543 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:35.584800 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:35.584802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:35.584803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:35.584965 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:35.585072 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:35.585426 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 02:07:35.585549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 02:07:35.587191 6811 tablet_service.cc:2908] Rejecting scan request for tablet 64a516afde4a40a98f93a6e7913f4d0d: Uninitialized: clean time has not yet been initialized
I20260808 02:07:35.589718 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6757
I20260808 02:07:35.601774 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6929
I20260808 02:07:35.614069 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 7094
I20260808 02:07:35.627532 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 6641
2026-08-08T02:07:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2714 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[7320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[7320](info): set up 2 sockets
Aug 08 02:07:35 dist-test-slave-bw88 krb5kdc[7320](info): commencing operation
krb5kdc: starting...
W20260808 02:07:36.849287 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:36 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:36.977491 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33629
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:37.100091 7338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:37.100369 7338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:37.100410 7338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:37.104667 7338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:37.104729 7338 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:07:37.104746 7338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:37.104794 7338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:37.104825 7338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:37.104857 7338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:37.110392 7338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33629
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:37.111789 7338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:37.113020 7338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:37.122159 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:37.122210 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:37.122179 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.122707 7338 server_base.cc:1034] running on GCE node
I20260808 02:07:37.123342 7338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:37.124495 7338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:37.125715 7338 hybrid_clock.cc:648] HybridClock initialized: now 1786154857125701 us; error 28 us; skew 500 ppm
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:37.128875 7338 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:37.129927 7338 webserver.cc:460] Webserver started at http://127.5.209.126:46261/ using document root <none> and password file <none>
I20260808 02:07:37.130532 7338 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:37.130627 7338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:37.130879 7338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:37.133000 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "b8a3e553b64844f5acb7fc0433235aea"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "b0a2c602254351fe267764c97913743b"
server_key_iv: "eb461715ec63288bf3e675a9e339c5d2"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.133637 7338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "b8a3e553b64844f5acb7fc0433235aea"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "b0a2c602254351fe267764c97913743b"
server_key_iv: "eb461715ec63288bf3e675a9e339c5d2"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.137466 7338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:37.140060 7357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.141618 7338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:37.141789 7338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "b8a3e553b64844f5acb7fc0433235aea"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "b0a2c602254351fe267764c97913743b"
server_key_iv: "eb461715ec63288bf3e675a9e339c5d2"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.141917 7338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:37.159902 7338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:37.163674 7338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:37.163928 7338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:37.173038 7442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33629 every 8 connection(s)
I20260808 02:07:37.173035 7338 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33629
I20260808 02:07:37.174152 7338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:37.177050 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:37.182744 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Bootstrap starting.
I20260808 02:07:37.184322 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 7338
I20260808 02:07:37.184448 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:37.184720 5957 external_mini_cluster.cc:1428] Setting key 9a88ec280f697bd40c5d4ee353395e11
I20260808 02:07:37.185549 7443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:37.186517 7443 log.cc:824] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:37.188709 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: No bootstrap required, opened a new log
I20260808 02:07:37.192785 7443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } }
I20260808 02:07:37.193125 7443 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:37.193182 7443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a3e553b64844f5acb7fc0433235aea, State: Initialized, Role: FOLLOWER
I20260808 02:07:37.193735 7443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } }
I20260808 02:07:37.193928 7443 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:37.194018 7443 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:37.194166 7443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 0 FOLLOWER]: Advancing to term 1
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:37.195079 7443 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } }
I20260808 02:07:37.195479 7443 leader_election.cc:302] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8a3e553b64844f5acb7fc0433235aea; no voters:
I20260808 02:07:37.195947 7443 leader_election.cc:288] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:37.196058 7448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:37.196259 7448 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [term 1 LEADER]: Becoming Leader. State: Replica: b8a3e553b64844f5acb7fc0433235aea, State: Running, Role: LEADER
I20260808 02:07:37.196630 7448 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } }
I20260808 02:07:37.197299 7443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:37.198364 7449 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8a3e553b64844f5acb7fc0433235aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } } }
I20260808 02:07:37.198515 7449 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:37.198856 7451 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8a3e553b64844f5acb7fc0433235aea. Latest consensus state: current_term: 1 leader_uuid: "b8a3e553b64844f5acb7fc0433235aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a3e553b64844f5acb7fc0433235aea" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33629 } } }
I20260808 02:07:37.199040 7451 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:37.198951 7458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:37.201411 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.185948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.5.209.126:33629)
0808 02:07:37.186443 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.186466 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.186507 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:07:37.187475 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.187496 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.187827 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.188142 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.189040 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.190186 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.191515 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.192085 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.192234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.195392 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.195398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.195414 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.195444 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.197652 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.198202 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.198211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.198220 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.198385 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.198754 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.198758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.198760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.199349 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.199611 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.199907 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.200394 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":147,"threads_started":1}
I20260808 02:07:37.203423 7458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:37.209555 7458 catalog_manager.cc:1347] Generated new cluster ID: 6962b753d3c14836badbb66187aa37ab
I20260808 02:07:37.209623 7458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:37.223904 7458 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:37.224797 7458 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:37.232865 7458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Generated new TSK 0
I20260808 02:07:37.233669 7458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:37.268071 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33629
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:37.402482 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:37.402736 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:37.402779 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:37.406823 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:37.406876 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:37.406914 7478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:37.406996 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:37.412125 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:37.413133 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:37.414198 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:37.424240 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.424348 7478 server_base.cc:1034] running on GCE node
W20260808 02:07:37.424284 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:37.424294 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.424930 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:37.425645 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:37.426842 7478 hybrid_clock.cc:648] HybridClock initialized: now 1786154857426828 us; error 28 us; skew 500 ppm
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:37.429632 7478 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:37.430460 7478 webserver.cc:460] Webserver started at http://127.5.209.65:44533/ using document root <none> and password file <none>
I20260808 02:07:37.430996 7478 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:37.431054 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:37.431219 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:37.433240 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "7c47a923bad0441aae43ef7056fa4718"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "351b927cd0b379a964659fdfc0c88ad7"
server_key_iv: "f6e19aaf6bc3e2d17584a3bb8efca594"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.433862 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "7c47a923bad0441aae43ef7056fa4718"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "351b927cd0b379a964659fdfc0c88ad7"
server_key_iv: "f6e19aaf6bc3e2d17584a3bb8efca594"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.437577 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:07:37.439996 7499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.441275 7478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:37.441485 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "7c47a923bad0441aae43ef7056fa4718"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "351b927cd0b379a964659fdfc0c88ad7"
server_key_iv: "f6e19aaf6bc3e2d17584a3bb8efca594"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.441599 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:37.461973 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:37.465097 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:37.465315 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:37.466977 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:37.467072 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.467163 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:37.467217 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:37.478974 7478 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:46865
I20260808 02:07:37.479036 7641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:46865 every 8 connection(s)
I20260808 02:07:37.479899 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:37.481599 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.467827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37239 (local address 127.5.209.126:33629)
0808 02:07:37.468006 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.468013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.469363 (+ 1350us) server_negotiation.cc:407] Connection header received
0808 02:07:37.470628 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.470634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.470707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.470824 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.472386 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.473312 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.474574 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.474887 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.474989 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.477873 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.477877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.477880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.477929 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.479503 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.480013 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.480017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.480020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.480110 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.480493 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.480496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.480498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.480719 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.480814 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.481327 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.481480 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:07:37.482378 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.468699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.65:37239)
0808 02:07:37.469173 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.469223 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.470369 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.471031 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.471038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.471482 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.472194 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.472214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.473448 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.473453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.474449 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.474459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.475055 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.475062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.475300 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.475865 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.475890 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.477712 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.479626 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.479633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.479647 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.479907 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.480206 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.480209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.480211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.480370 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.480843 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.480849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.481096 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.481275 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.481589 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260808 02:07:37.483759 7642 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33629
I20260808 02:07:37.483925 7642 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:37.484418 7642 heartbeater.cc:511] Master 127.5.209.126:33629 requested a full tablet report, sending...
I20260808 02:07:37.486104 7380 ts_manager.cc:194] Registered new tserver with Master: 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865)
I20260808 02:07:37.486804 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 7478
I20260808 02:07:37.486929 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:37.487236 5957 external_mini_cluster.cc:1428] Setting key 1f31b856fa9953834e4fb5f5eae2a0fd
I20260808 02:07:37.487551 7380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:37239
I20260808 02:07:37.493777 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.485974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:60053 (local address 127.5.209.126:33629)
0808 02:07:37.486097 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.486107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.486349 (+ 242us) server_negotiation.cc:407] Connection header received
0808 02:07:37.486453 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.486456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.486496 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.486588 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.487078 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.487819 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.488856 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.489190 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.489312 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.490693 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.490696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.490698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.490719 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.492110 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.492626 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.492629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.492630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.492703 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.492999 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.493001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.493002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.493166 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.493242 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.493578 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.493673 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:07:37.494009 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.485855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.65:60053)
0808 02:07:37.486263 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.486277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.486373 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.486606 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.486609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.486809 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.486925 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.486932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.487953 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.487956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.488728 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.488739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.489408 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.489414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.489547 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.490057 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.490077 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.490543 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.492238 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.492244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.492250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.492521 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.492790 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.492796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.492801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.492906 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.493261 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.493266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.493400 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.493519 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.493715 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:37.516649 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33629
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:37.631685 7655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:37.631930 7655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:37.631973 7655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:37.635937 7655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:37.635996 7655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:37.636034 7655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:37.636066 7655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:37.641527 7655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:37.642562 7655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:37.643572 7655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:37.654582 7668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:37.654592 7670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.654759 7655 server_base.cc:1034] running on GCE node
W20260808 02:07:37.654592 7667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.655313 7655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:37.655946 7655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:37.657131 7655 hybrid_clock.cc:648] HybridClock initialized: now 1786154857657116 us; error 24 us; skew 500 ppm
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:37.659958 7655 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:37.660804 7655 webserver.cc:460] Webserver started at http://127.5.209.66:46337/ using document root <none> and password file <none>
I20260808 02:07:37.661442 7655 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:37.661550 7655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:37.661782 7655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:37.663808 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "06114dcc94f54a5bbe4d2ef9c93981c8"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "2a26b57c5d10861ea375f8cd0b6b5ad6"
server_key_iv: "81a6dd90c5eb333d885a86bf3e1d5a06"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.664381 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "06114dcc94f54a5bbe4d2ef9c93981c8"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "2a26b57c5d10861ea375f8cd0b6b5ad6"
server_key_iv: "81a6dd90c5eb333d885a86bf3e1d5a06"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.668066 7655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 02:07:37.670518 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.671674 7655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:37.671806 7655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "06114dcc94f54a5bbe4d2ef9c93981c8"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "2a26b57c5d10861ea375f8cd0b6b5ad6"
server_key_iv: "81a6dd90c5eb333d885a86bf3e1d5a06"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.671912 7655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:37.713819 7655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:37.717186 7655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:37.717445 7655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:37.719039 7655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:37.719116 7655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.719184 7655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:37.719230 7655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:37.730619 7655 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:35479
I20260808 02:07:37.730648 7816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:35479 every 8 connection(s)
I20260808 02:07:37.731781 7655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:37.733232 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.719827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:58453 (local address 127.5.209.126:33629)
0808 02:07:37.719956 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.719960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.721013 (+ 1053us) server_negotiation.cc:407] Connection header received
0808 02:07:37.722025 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.722028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.722086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.722170 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.723795 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.724694 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.726083 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.726311 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.726445 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.729136 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.729139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.729142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.729168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.730928 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.731426 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.731433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.731435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.731533 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.731880 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.731883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.731885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.732164 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.732290 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.732945 (+ 655us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.733047 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:07:37.733566 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 7655
I20260808 02:07:37.733678 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:37.733953 5957 external_mini_cluster.cc:1428] Setting key 000c9f56773aac34895fd2e7214170fc
I20260808 02:07:37.734337 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.720394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.66:58453)
0808 02:07:37.720870 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.720911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.721815 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.722345 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.722354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.722850 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.723536 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.723615 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.724858 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.724861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.725949 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.725960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.726526 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.726532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.726778 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.727496 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.727513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.728952 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.731048 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.731052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.731063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.731315 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.731645 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.731647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.731648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.731775 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.732347 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.732354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.732684 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.732893 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.733284 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260808 02:07:37.735822 7817 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33629
I20260808 02:07:37.735985 7817 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:37.736565 7817 heartbeater.cc:511] Master 127.5.209.126:33629 requested a full tablet report, sending...
I20260808 02:07:37.737810 7381 ts_manager.cc:194] Registered new tserver with Master: 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479)
I20260808 02:07:37.738591 7381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:58453
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:07:37.747103 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.738108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.66:44451)
0808 02:07:37.738420 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.738434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.738517 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.739253 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.739256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.739457 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.739623 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.739631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.740710 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.740713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.741279 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.741287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.741995 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.741998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.742073 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.742627 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.742639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.742964 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.745060 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.745066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.745068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.745288 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.745682 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.745685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.745687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.745855 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.746526 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.746529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.746618 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.746704 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.746890 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260808 02:07:37.747151 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.738926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:44451 (local address 127.5.209.126:33629)
0808 02:07:37.739041 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.739045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.739060 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:07:37.739106 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.739109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.739142 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.739233 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.739743 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.740549 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.741452 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.741844 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.741965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.743136 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.743143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.743148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.743183 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.744803 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.745440 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.745447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.745452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.745549 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.745983 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.745987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.745990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.746405 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.746504 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.746811 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.746945 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:37.768800 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33629
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:37.886024 7832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:37.886314 7832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:37.886400 7832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:37.890434 7832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:37.890511 7832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:37.890579 7832 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:37.890661 7832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:37.895936 7832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:37.897037 7832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:37.898273 7832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:37.908540 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:37.908715 7845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.908751 7832 server_base.cc:1034] running on GCE node
W20260808 02:07:37.908543 7843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:37.909296 7832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:37.910058 7832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:37.911235 7832 hybrid_clock.cc:648] HybridClock initialized: now 1786154857911206 us; error 39 us; skew 500 ppm
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:37.914227 7832 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:37.915150 7832 webserver.cc:460] Webserver started at http://127.5.209.67:35927/ using document root <none> and password file <none>
I20260808 02:07:37.915688 7832 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:37.915747 7832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:37.915915 7832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:37.918119 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "dd43071d2a3745fdadd5812f273d476c"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "8a9f87489d462729c8ba2ce3ab57a99d"
server_key_iv: "679677492944f3fa891e8338f08ea5f5"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.918653 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "dd43071d2a3745fdadd5812f273d476c"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "8a9f87489d462729c8ba2ce3ab57a99d"
server_key_iv: "679677492944f3fa891e8338f08ea5f5"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.922454 7832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:07:37.924837 7854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.926203 7832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:07:37.926342 7832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "dd43071d2a3745fdadd5812f273d476c"
format_stamp: "Formatted at 2026-08-08 02:07:37 on dist-test-slave-bw88"
server_key: "8a9f87489d462729c8ba2ce3ab57a99d"
server_key_iv: "679677492944f3fa891e8338f08ea5f5"
server_key_version: "encryptionkey@0"
I20260808 02:07:37.926463 7832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:37.939911 7832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:37.943413 7832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:37.943645 7832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:37.945210 7832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:37.945304 7832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:37.945425 7832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:37.945483 7832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:37.957571 7832 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:33807
I20260808 02:07:37.958776 7832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:37.959576 8006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:33807 every 8 connection(s)
I20260808 02:07:37.961735 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.946397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59319 (local address 127.5.209.126:33629)
0808 02:07:37.946577 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.946584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.947620 (+ 1036us) server_negotiation.cc:407] Connection header received
0808 02:07:37.949030 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.949035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.949108 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.949177 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.950878 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.951600 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.953162 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.953594 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.953734 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.957560 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.957565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.957568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.957602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.959167 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.959870 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.959874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.959876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.959981 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.960423 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.960426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.960428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.960654 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.960785 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.961388 (+ 603us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.961541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:07:37.962589 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.946876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.67:59319)
0808 02:07:37.947431 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.947488 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.948701 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.949663 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.949670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.950035 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.950715 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.950730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.951747 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.951751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.953007 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.953018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.953785 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.953794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.954055 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.955413 (+ 1358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.955430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.957291 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.959337 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.959344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.959360 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.959758 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.960099 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.960103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.960106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.960288 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.960785 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.960794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.961073 (+ 279us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.961300 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.961699 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":177,"threads_started":1}
I20260808 02:07:37.964038 8007 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33629
I20260808 02:07:37.964216 8007 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:37.964759 8007 heartbeater.cc:511] Master 127.5.209.126:33629 requested a full tablet report, sending...
I20260808 02:07:37.965904 7380 ts_manager.cc:194] Registered new tserver with Master: dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807)
I20260808 02:07:37.966202 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 7832
I20260808 02:07:37.966351 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:37.966674 7380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:59319
I20260808 02:07:37.966715 5957 external_mini_cluster.cc:1428] Setting key a0b5ad62b76c0d03e29006c9817d83b7
I20260808 02:07:37.969082 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:07:37.974627 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.966431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.5.209.67:52803)
0808 02:07:37.966773 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:37.966788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:37.966891 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:37.967189 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:37.967192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:37.967414 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:37.967553 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.967561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.968680 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.968684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.969509 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:37.969519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.969988 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.969992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:37.970082 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.970599 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:37.970627 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:37.970916 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:37.972967 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.972970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.972971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.973265 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.973601 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:37.973603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:37.973605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:37.973703 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:37.974182 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:37.974184 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:37.974269 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 02:07:37.974364 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:07:37.974489 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260808 02:07:37.974877 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.966816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:52803 (local address 127.5.209.126:33629)
0808 02:07:37.966933 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.966937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.966966 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:07:37.967005 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.967008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.967070 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.967144 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.967701 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.968519 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.969650 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.969873 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.969982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.971053 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.971056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.971059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.971110 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.972827 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.973395 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.973402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.973407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.973499 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.973799 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.973804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.973809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.974061 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.974178 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.974453 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.974591 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:07:37 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:37.998911 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:37.989751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.5.209.126:33629)
0808 02:07:37.989895 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:37.989899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:37.989913 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:07:37.989966 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:37.989969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:37.990023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:37.990107 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:37.990767 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.991306 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.992166 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:37.992620 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:37.992769 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:37.995298 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:37.995301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:37.995303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:37.995329 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:37.997180 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.997664 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.997667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.997668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.997768 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:37.998070 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:37.998072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:37.998074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:37.998249 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:37.998354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:37.998581 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 02:07:37.998714 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:07:38.001924 7380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:38.004884 7380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:38.019526 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.013306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:35479 (local address 127.0.0.1:50482)
0808 02:07:38.014127 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.014178 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.014330 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.015364 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.015367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.015386 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.015494 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.015504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.016542 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.016546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.017947 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.017958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.018962 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.018966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.019068 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.019115 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.019224 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.019360 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":61,"spinlock_wait_cycles":4992,"thread_start_us":104,"threads_started":1}
I20260808 02:07:38.019570 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.013714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33807 (local address 127.0.0.1:48102)
0808 02:07:38.014583 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.014597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.014704 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.015546 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.015552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.015573 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.015663 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.015672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.017012 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.017016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.018174 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.018184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.019212 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.019218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.019315 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.019336 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.019407 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.019462 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260808 02:07:38.019716 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.013969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.5.209.67:33807)
0808 02:07:38.014422 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.014436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.015111 (+ 675us) server_negotiation.cc:407] Connection header received
0808 02:07:38.015195 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.015199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.015388 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.015543 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.015787 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.016890 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.018299 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.019057 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.019164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.019339 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.019446 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.019545 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":118,"threads_started":1}
I20260808 02:07:38.020126 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.013468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.5.209.66:35479)
0808 02:07:38.014917 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.014926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.014944 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:07:38.015031 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.015036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.015231 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.015393 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.015628 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.016426 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.018083 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.018839 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.018962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.019130 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.019876 (+ 746us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.019978 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":125,"threads_started":1}
I20260808 02:07:38.020375 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.013422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46865 (local address 127.0.0.1:41528)
0808 02:07:38.014126 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.014177 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.014330 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.015158 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.015165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.015188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.015321 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.015328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.016982 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.016986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.018430 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.018440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.019513 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.019517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.020058 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.020086 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.020142 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.020233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"spinlock_wait_cycles":10752,"thread_start_us":117,"threads_started":1}
I20260808 02:07:38.021267 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.014148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.5.209.65:46865)
0808 02:07:38.014623 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.014628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.014671 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:07:38.014754 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.014760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.014981 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.015498 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.015638 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.016841 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.018572 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.019355 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.020613 (+ 1258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.020764 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.020814 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.020904 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":102,"threads_started":1}
I20260808 02:07:38.022965 7732 tablet_service.cc:1467] Processing CreateTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab (DEFAULT_TABLE table=test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.022948 7922 tablet_service.cc:1467] Processing CreateTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab (DEFAULT_TABLE table=test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.023466 7564 tablet_service.cc:1467] Processing CreateTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab (DEFAULT_TABLE table=test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.024140 7922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35db00acd0ba4f039f5ea7553ea8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.024140 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35db00acd0ba4f039f5ea7553ea8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.024490 7564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35db00acd0ba4f039f5ea7553ea8f4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.032469 8039 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Bootstrap starting.
I20260808 02:07:38.033049 8040 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Bootstrap starting.
I20260808 02:07:38.033676 8038 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Bootstrap starting.
I20260808 02:07:38.034778 8039 tablet_bootstrap.cc:654] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.035264 8040 tablet_bootstrap.cc:654] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.035717 8038 tablet_bootstrap.cc:654] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.035758 8039 log.cc:824] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:38.036085 8040 log.cc:824] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:38.036523 8038 log.cc:824] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:38.038307 8038 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: No bootstrap required, opened a new log
I20260808 02:07:38.038368 8040 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: No bootstrap required, opened a new log
I20260808 02:07:38.038367 8039 tablet_bootstrap.cc:492] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: No bootstrap required, opened a new log
I20260808 02:07:38.038537 8038 ts_tablet_manager.cc:1397] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:07:38.038552 8039 ts_tablet_manager.cc:1397] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:07:38.038553 8040 ts_tablet_manager.cc:1397] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:07:38.042253 8038 raft_consensus.cc:348] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.042547 8038 raft_consensus.cc:374] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.042618 8038 raft_consensus.cc:729] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c47a923bad0441aae43ef7056fa4718, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.042640 8039 raft_consensus.cc:348] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.042635 8040 raft_consensus.cc:348] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.042873 8039 raft_consensus.cc:374] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.042873 8040 raft_consensus.cc:374] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.042934 8040 raft_consensus.cc:729] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06114dcc94f54a5bbe4d2ef9c93981c8, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.042934 8039 raft_consensus.cc:729] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd43071d2a3745fdadd5812f273d476c, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.043145 8038 consensus_queue.cc:260] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.043406 8039 consensus_queue.cc:260] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.043406 8040 consensus_queue.cc:260] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.043900 7642 heartbeater.cc:503] Master 127.5.209.126:33629 was elected leader, sending a full tablet report...
I20260808 02:07:38.044425 8039 ts_tablet_manager.cc:1428] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:38.044425 8040 ts_tablet_manager.cc:1428] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:38.044497 7817 heartbeater.cc:503] Master 127.5.209.126:33629 was elected leader, sending a full tablet report...
I20260808 02:07:38.044857 8007 heartbeater.cc:503] Master 127.5.209.126:33629 was elected leader, sending a full tablet report...
I20260808 02:07:38.046044 8038 ts_tablet_manager.cc:1428] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260808 02:07:38.058543 8046 raft_consensus.cc:482] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:38.058705 8046 raft_consensus.cc:504] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.059726 8046 leader_election.cc:288] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865), dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807)
I20260808 02:07:38.064252 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.060069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:34349 (local address 127.5.209.65:46865)
0808 02:07:38.060196 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.060201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.060641 (+ 440us) server_negotiation.cc:407] Connection header received
0808 02:07:38.060800 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.060803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.060848 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.060924 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.061292 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.061980 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.063032 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.063558 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.063686 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.063739 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.064019 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.064100 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:07:38.064282 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.059968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46865 (local address 127.5.209.66:34349)
0808 02:07:38.060526 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.060543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.060692 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.060960 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.060963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.060983 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.061138 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.061157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.062079 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.062084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.062903 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.062913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.063702 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.063705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.063792 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.063815 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.063987 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.064103 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":48,"spinlock_wait_cycles":28416,"thread_start_us":109,"threads_started":1}
I20260808 02:07:38.064654 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.059968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33807 (local address 127.5.209.66:35991)
0808 02:07:38.060195 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.060211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.060314 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.060636 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.060639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.060693 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.060831 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.060839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.061976 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.061979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.063316 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.063327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.064197 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.064200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.064310 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.064335 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.064415 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.064535 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"spinlock_wait_cycles":15360}
I20260808 02:07:38.064654 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.060055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:35991 (local address 127.5.209.67:33807)
0808 02:07:38.060196 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.060201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.060286 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:07:38.060436 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.060439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.060498 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.060605 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.060971 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.061815 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.063446 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.064060 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.064184 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.064228 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.064455 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.064535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:07:38.065050 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35db00acd0ba4f039f5ea7553ea8f4ab" candidate_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c47a923bad0441aae43ef7056fa4718" is_pre_election: true
I20260808 02:07:38.065222 7947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35db00acd0ba4f039f5ea7553ea8f4ab" candidate_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd43071d2a3745fdadd5812f273d476c" is_pre_election: true
I20260808 02:07:38.065421 7592 raft_consensus.cc:2454] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06114dcc94f54a5bbe4d2ef9c93981c8 in term 0.
I20260808 02:07:38.065551 7947 raft_consensus.cc:2454] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06114dcc94f54a5bbe4d2ef9c93981c8 in term 0.
I20260808 02:07:38.065934 7679 leader_election.cc:302] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06114dcc94f54a5bbe4d2ef9c93981c8, 7c47a923bad0441aae43ef7056fa4718; no voters:
I20260808 02:07:38.066227 8046 raft_consensus.cc:2781] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:38.066326 8046 raft_consensus.cc:482] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:38.066388 8046 raft_consensus.cc:3037] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.067476 8046 raft_consensus.cc:504] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.067828 8046 leader_election.cc:288] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865), dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807)
I20260808 02:07:38.068161 7947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35db00acd0ba4f039f5ea7553ea8f4ab" candidate_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd43071d2a3745fdadd5812f273d476c"
I20260808 02:07:38.068291 7947 raft_consensus.cc:3037] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.069358 7947 raft_consensus.cc:2454] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06114dcc94f54a5bbe4d2ef9c93981c8 in term 1.
I20260808 02:07:38.069517 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35db00acd0ba4f039f5ea7553ea8f4ab" candidate_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c47a923bad0441aae43ef7056fa4718"
I20260808 02:07:38.069643 7592 raft_consensus.cc:3037] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.069711 7682 leader_election.cc:302] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06114dcc94f54a5bbe4d2ef9c93981c8, dd43071d2a3745fdadd5812f273d476c; no voters:
I20260808 02:07:38.069927 8046 raft_consensus.cc:2781] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:38.070168 8046 raft_consensus.cc:686] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 LEADER]: Becoming Leader. State: Replica: 06114dcc94f54a5bbe4d2ef9c93981c8, State: Running, Role: LEADER
I20260808 02:07:38.070506 8046 consensus_queue.cc:237] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.071210 7592 raft_consensus.cc:2454] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06114dcc94f54a5bbe4d2ef9c93981c8 in term 1.
I20260808 02:07:38.074179 7374 catalog_manager.cc:5495] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:38.104214 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.099813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.5.209.66:35479)
0808 02:07:38.099972 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.099977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.099992 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:07:38.100223 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.100226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.100282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.100386 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:38.100668 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.101195 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.102656 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.102972 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.103064 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.103365 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.103771 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.103862 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.103938 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.104002 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:07:38.110253 7947 raft_consensus.cc:1261] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Refusing update from remote peer 06114dcc94f54a5bbe4d2ef9c93981c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.110255 7592 raft_consensus.cc:1261] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Refusing update from remote peer 06114dcc94f54a5bbe4d2ef9c93981c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.110953 8046 consensus_queue.cc:1035] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.111200 8050 consensus_queue.cc:1035] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.121380 8056 mvcc.cc:205] Tried to move back new op lower bound from 7316090298812219392 to 7316090298660290560. Current Snapshot: MvccSnapshot[applied={T|T < 7316090298812219392}]
I20260808 02:07:38.133064 7374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:38.133304 7374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:38.136405 7374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Sending DeleteTablet for 3 replicas of tablet 35db00acd0ba4f039f5ea7553ea8f4ab
I20260808 02:07:38.137398 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:50482
I20260808 02:07:38.137482 7564 tablet_service.cc:1514] Processing DeleteTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:41528
I20260808 02:07:38.137476 7922 tablet_service.cc:1514] Processing DeleteTablet for tablet 35db00acd0ba4f039f5ea7553ea8f4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48102
I20260808 02:07:38.138239 8065 tablet_replica.cc:330] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: stopping tablet replica
I20260808 02:07:38.138267 8066 tablet_replica.cc:330] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: stopping tablet replica
I20260808 02:07:38.138609 8065 raft_consensus.cc:2229] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:38.138610 8066 raft_consensus.cc:2229] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:38.139045 8066 raft_consensus.cc:2258] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.139099 8065 raft_consensus.cc:2258] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.139163 7374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:07:38.139616 8064 tablet_replica.cc:330] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: stopping tablet replica
W20260808 02:07:38.139709 7374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:38.139958 8064 raft_consensus.cc:2229] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:38.140275 8064 raft_consensus.cc:2258] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.141475 8065 ts_tablet_manager.cc:1905] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.141752 8066 ts_tablet_manager.cc:1905] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.145001 7564 tablet_service.cc:1467] Processing CreateTablet for tablet b641ebee336a4005ac9a37cc57093406 (DEFAULT_TABLE table=test-table [id=dc30477cbd9e40c09391ece0c714ac55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.145367 7564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b641ebee336a4005ac9a37cc57093406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.145861 8064 ts_tablet_manager.cc:1905] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.148355 8038 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Bootstrap starting.
I20260808 02:07:38.148890 7732 tablet_service.cc:1467] Processing CreateTablet for tablet b641ebee336a4005ac9a37cc57093406 (DEFAULT_TABLE table=test-table [id=dc30477cbd9e40c09391ece0c714ac55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.150552 8038 tablet_bootstrap.cc:654] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.150652 7922 tablet_service.cc:1467] Processing CreateTablet for tablet b641ebee336a4005ac9a37cc57093406 (DEFAULT_TABLE table=test-table [id=dc30477cbd9e40c09391ece0c714ac55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:38.150967 8064 ts_tablet_manager.cc:1918] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:38.151057 8064 log.cc:1196] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/35db00acd0ba4f039f5ea7553ea8f4ab
I20260808 02:07:38.151059 7922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b641ebee336a4005ac9a37cc57093406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.151381 8064 ts_tablet_manager.cc:1939] T 35db00acd0ba4f039f5ea7553ea8f4ab P 7c47a923bad0441aae43ef7056fa4718: Deleting consensus metadata
I20260808 02:07:38.153446 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b641ebee336a4005ac9a37cc57093406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.153544 7358 catalog_manager.cc:4841] TS 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865): tablet 35db00acd0ba4f039f5ea7553ea8f4ab (table test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]) successfully deleted
I20260808 02:07:38.154054 8039 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Bootstrap starting.
I20260808 02:07:38.155383 8039 tablet_bootstrap.cc:654] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.150704 8065 ts_tablet_manager.cc:1918] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:38.156327 8065 log.cc:1196] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/35db00acd0ba4f039f5ea7553ea8f4ab
I20260808 02:07:38.156757 8040 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Bootstrap starting.
I20260808 02:07:38.156761 8065 ts_tablet_manager.cc:1939] T 35db00acd0ba4f039f5ea7553ea8f4ab P dd43071d2a3745fdadd5812f273d476c: Deleting consensus metadata
I20260808 02:07:38.158655 8040 tablet_bootstrap.cc:654] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.158856 7361 catalog_manager.cc:4841] TS dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807): tablet 35db00acd0ba4f039f5ea7553ea8f4ab (table test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]) successfully deleted
I20260808 02:07:38.160023 8038 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: No bootstrap required, opened a new log
I20260808 02:07:38.160132 8038 ts_tablet_manager.cc:1397] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260808 02:07:38.160539 8066 ts_tablet_manager.cc:1918] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:38.160727 8039 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: No bootstrap required, opened a new log
I20260808 02:07:38.160751 8066 log.cc:1196] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/35db00acd0ba4f039f5ea7553ea8f4ab
I20260808 02:07:38.160816 8039 ts_tablet_manager.cc:1397] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260808 02:07:38.161177 8066 ts_tablet_manager.cc:1939] T 35db00acd0ba4f039f5ea7553ea8f4ab P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting consensus metadata
I20260808 02:07:38.161177 8038 raft_consensus.cc:348] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.161311 8038 raft_consensus.cc:374] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.161386 8038 raft_consensus.cc:729] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c47a923bad0441aae43ef7056fa4718, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.161545 8038 consensus_queue.cc:260] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.161823 8038 ts_tablet_manager.cc:1428] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:38.161775 8039 raft_consensus.cc:348] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.161891 8039 raft_consensus.cc:374] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.161937 8039 raft_consensus.cc:729] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd43071d2a3745fdadd5812f273d476c, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.162065 8039 consensus_queue.cc:260] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.162302 8039 ts_tablet_manager.cc:1428] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 02:07:38.162685 7360 catalog_manager.cc:4841] TS 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479): tablet 35db00acd0ba4f039f5ea7553ea8f4ab (table test-table [id=26cc5e3ffa3a4910a50777b3884a22b6]) successfully deleted
I20260808 02:07:38.164590 8040 tablet_bootstrap.cc:492] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: No bootstrap required, opened a new log
I20260808 02:07:38.164682 8040 ts_tablet_manager.cc:1397] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260808 02:07:38.165536 8040 raft_consensus.cc:348] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.165653 8040 raft_consensus.cc:374] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.165683 8040 raft_consensus.cc:729] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06114dcc94f54a5bbe4d2ef9c93981c8, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.165804 8040 consensus_queue.cc:260] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.166054 8040 ts_tablet_manager.cc:1428] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:38.179628 8045 raft_consensus.cc:482] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:38.179787 8045 raft_consensus.cc:504] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.181224 8045 leader_election.cc:288] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479), dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807)
I20260808 02:07:38.187686 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.181534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:35627 (local address 127.5.209.66:35479)
0808 02:07:38.181681 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.181685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.182140 (+ 455us) server_negotiation.cc:407] Connection header received
0808 02:07:38.182333 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.182338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.182395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.182443 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.182959 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.183754 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.185948 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.186722 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.186811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.186863 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.187490 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.187567 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:07:38.188143 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.181580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:35479 (local address 127.5.209.65:35627)
0808 02:07:38.182056 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.182071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.182211 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.182586 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.182589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.182604 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.182827 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.182833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.183899 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.183902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.185798 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.185809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.187085 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.187090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.187296 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.187324 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.187857 (+ 533us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.187943 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":168,"threads_started":1}
I20260808 02:07:38.188221 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.181845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:40393 (local address 127.5.209.67:33807)
0808 02:07:38.181989 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.181995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.182114 (+ 119us) server_negotiation.cc:407] Connection header received
0808 02:07:38.182310 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.182314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.182398 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.182482 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.183227 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.184051 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.186583 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.187208 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.187299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.187345 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.188021 (+ 676us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.188102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:07:38.188890 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.181556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33807 (local address 127.5.209.65:40393)
0808 02:07:38.181989 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.182006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.182191 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.182716 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.182718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.182737 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.183048 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.183057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.184242 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.184245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.186423 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.186446 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.187510 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.187514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.187677 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.187702 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.188653 (+ 951us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.188739 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":89,"threads_started":1}
I20260808 02:07:38.188884 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b641ebee336a4005ac9a37cc57093406" candidate_uuid: "7c47a923bad0441aae43ef7056fa4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" is_pre_election: true
I20260808 02:07:38.189134 7758 raft_consensus.cc:2454] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c47a923bad0441aae43ef7056fa4718 in term 0.
I20260808 02:07:38.189318 7947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b641ebee336a4005ac9a37cc57093406" candidate_uuid: "7c47a923bad0441aae43ef7056fa4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd43071d2a3745fdadd5812f273d476c" is_pre_election: true
I20260808 02:07:38.189491 7947 raft_consensus.cc:2454] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c47a923bad0441aae43ef7056fa4718 in term 0.
I20260808 02:07:38.189704 7502 leader_election.cc:302] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06114dcc94f54a5bbe4d2ef9c93981c8, 7c47a923bad0441aae43ef7056fa4718; no voters:
I20260808 02:07:38.189994 8045 raft_consensus.cc:2781] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:38.190063 8045 raft_consensus.cc:482] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:38.190104 8045 raft_consensus.cc:3037] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.191674 8045 raft_consensus.cc:504] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.192207 8045 leader_election.cc:288] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [CANDIDATE]: Term 1 election: Requested vote from peers 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479), dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807)
I20260808 02:07:38.192531 7947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b641ebee336a4005ac9a37cc57093406" candidate_uuid: "7c47a923bad0441aae43ef7056fa4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd43071d2a3745fdadd5812f273d476c"
I20260808 02:07:38.192649 7947 raft_consensus.cc:3037] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.192741 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b641ebee336a4005ac9a37cc57093406" candidate_uuid: "7c47a923bad0441aae43ef7056fa4718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8"
I20260808 02:07:38.192878 7758 raft_consensus.cc:3037] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.193926 7947 raft_consensus.cc:2454] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c47a923bad0441aae43ef7056fa4718 in term 1.
I20260808 02:07:38.194305 7503 leader_election.cc:302] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c47a923bad0441aae43ef7056fa4718, dd43071d2a3745fdadd5812f273d476c; no voters:
I20260808 02:07:38.194555 8045 raft_consensus.cc:2781] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:38.194796 8045 raft_consensus.cc:686] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 1 LEADER]: Becoming Leader. State: Replica: 7c47a923bad0441aae43ef7056fa4718, State: Running, Role: LEADER
I20260808 02:07:38.194868 7758 raft_consensus.cc:2454] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c47a923bad0441aae43ef7056fa4718 in term 1.
I20260808 02:07:38.195066 8045 consensus_queue.cc:237] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.197414 7380 catalog_manager.cc:5495] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "7c47a923bad0441aae43ef7056fa4718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } health_report { overall_health: UNKNOWN } } }
W20260808 02:07:38.210089 8008 tablet.cc:2369] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:07:38.231449 7644 tablet.cc:2369] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:07:38.233424 7818 tablet.cc:2369] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:07:38 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:38.241106 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.231593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.5.209.126:33629)
0808 02:07:38.231799 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.231804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.232201 (+ 397us) server_negotiation.cc:407] Connection header received
0808 02:07:38.232366 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.232369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.232416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.232509 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:38.233213 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.233908 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.234720 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.234943 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.235139 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.237742 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:38.237745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:38.237747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:38.237770 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:38.239277 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.239827 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.239830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.239831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.239902 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.240197 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.240199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.240200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.240369 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:38.240444 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.240807 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.240954 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":112}
I20260808 02:07:38.241142 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.231508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33629 (local address 127.0.0.1:58494)
0808 02:07:38.232097 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.232111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.232265 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.232535 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.232537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.232978 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:38.233074 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.233081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.234032 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.234037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.234598 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.234606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.235062 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.235066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.235184 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.235820 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:38.235833 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:38.237594 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:38.239392 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:38.239402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:38.239410 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:38.239729 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:38.239999 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:38.240003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:38.240005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:38.240106 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:38.240473 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:38.240480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:38.240619 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.240742 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.240901 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":20,"thread_start_us":165,"threads_started":1}
I20260808 02:07:38.245687 7380 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:58494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:07:38.246610 7380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:38.251868 7564 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e6e3bb13045f4b91a203db8c577b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a494b4d7e5a4123b8c70e38cab73354]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:38.251942 7732 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e6e3bb13045f4b91a203db8c577b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a494b4d7e5a4123b8c70e38cab73354]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:38.252333 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e6e3bb13045f4b91a203db8c577b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.252344 7564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e6e3bb13045f4b91a203db8c577b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.254848 8040 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: Bootstrap starting.
I20260808 02:07:38.255863 7922 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e6e3bb13045f4b91a203db8c577b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a494b4d7e5a4123b8c70e38cab73354]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:38.256271 7922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e6e3bb13045f4b91a203db8c577b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:38.256335 8040 tablet_bootstrap.cc:654] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.256909 8038 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: Bootstrap starting.
I20260808 02:07:38.257555 8040 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: No bootstrap required, opened a new log
I20260808 02:07:38.257630 8040 ts_tablet_manager.cc:1397] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:38.258037 8040 raft_consensus.cc:348] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.258163 8040 raft_consensus.cc:374] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.258198 8040 raft_consensus.cc:729] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06114dcc94f54a5bbe4d2ef9c93981c8, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.258366 8040 consensus_queue.cc:260] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.258559 8038 tablet_bootstrap.cc:654] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.258677 8040 ts_tablet_manager.cc:1428] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:07:38.258617 8050 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.258744 8050 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.260033 8039 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: Bootstrap starting.
I20260808 02:07:38.261752 8039 tablet_bootstrap.cc:654] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:38.262543 8038 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: No bootstrap required, opened a new log
I20260808 02:07:38.262653 8038 ts_tablet_manager.cc:1397] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260808 02:07:38.263388 8038 raft_consensus.cc:348] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.263639 8038 raft_consensus.cc:374] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.263707 8038 raft_consensus.cc:729] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c47a923bad0441aae43ef7056fa4718, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.264194 8038 consensus_queue.cc:260] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.264279 8039 tablet_bootstrap.cc:492] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: No bootstrap required, opened a new log
I20260808 02:07:38.264434 8039 ts_tablet_manager.cc:1397] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260808 02:07:38.264567 8075 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.264735 8075 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.265200 8039 raft_consensus.cc:348] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.265322 8039 raft_consensus.cc:374] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:38.265422 8038 ts_tablet_manager.cc:1428] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 02:07:38.265419 8039 raft_consensus.cc:729] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd43071d2a3745fdadd5812f273d476c, State: Initialized, Role: FOLLOWER
I20260808 02:07:38.265611 8039 consensus_queue.cc:260] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.265967 8047 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.266198 8047 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.266537 8039 ts_tablet_manager.cc:1428] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:38.451483 8047 raft_consensus.cc:482] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:38.451673 8047 raft_consensus.cc:504] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.453135 8047 leader_election.cc:288] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865), 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479)
I20260808 02:07:38.457595 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.453486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:42739 (local address 127.5.209.65:46865)
0808 02:07:38.453617 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.453623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.453703 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:07:38.453820 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.453844 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.453910 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.453990 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.454443 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.455032 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.456278 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.456975 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.457120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.457178 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.457420 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.457493 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:07:38.457599 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.453374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46865 (local address 127.5.209.67:42739)
0808 02:07:38.453598 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.453615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.453717 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.454021 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.454024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.454045 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.454292 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.454302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.455150 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.455156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.456126 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.456136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.457111 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.457115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.457216 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.457239 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.457385 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.457465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260808 02:07:38.458097 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e6e3bb13045f4b91a203db8c577b5" candidate_uuid: "dd43071d2a3745fdadd5812f273d476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c47a923bad0441aae43ef7056fa4718" is_pre_election: true
I20260808 02:07:38.458295 7592 raft_consensus.cc:2454] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd43071d2a3745fdadd5812f273d476c in term 0.
I20260808 02:07:38.458699 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.453531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:52739 (local address 127.5.209.66:35479)
0808 02:07:38.453737 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.453743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.453936 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:07:38.454031 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.454034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.454079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.454160 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:38.454579 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.455121 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.457361 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.458020 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.458151 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.458201 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.458499 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.458592 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260808 02:07:38.458684 7856 leader_election.cc:302] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c47a923bad0441aae43ef7056fa4718, dd43071d2a3745fdadd5812f273d476c; no voters:
I20260808 02:07:38.458948 8047 raft_consensus.cc:2781] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:38.459019 8047 raft_consensus.cc:482] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:38.459074 8047 raft_consensus.cc:3037] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.459208 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.453416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:35479 (local address 127.5.209.67:52739)
0808 02:07:38.453758 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.453772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.453947 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.454188 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.454190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.454205 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:38.454456 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.454465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.455288 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.455291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.457179 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.457192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.458152 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.458155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.458326 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.458347 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.458910 (+ 563us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.458993 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":87,"threads_started":1}
I20260808 02:07:38.459695 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e6e3bb13045f4b91a203db8c577b5" candidate_uuid: "dd43071d2a3745fdadd5812f273d476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" is_pre_election: true
I20260808 02:07:38.459847 7758 raft_consensus.cc:2454] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd43071d2a3745fdadd5812f273d476c in term 0.
I20260808 02:07:38.460438 8047 raft_consensus.cc:504] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.460815 8047 leader_election.cc:288] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [CANDIDATE]: Term 1 election: Requested vote from peers 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865), 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479)
I20260808 02:07:38.461129 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e6e3bb13045f4b91a203db8c577b5" candidate_uuid: "dd43071d2a3745fdadd5812f273d476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c47a923bad0441aae43ef7056fa4718"
I20260808 02:07:38.461189 7758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e6e3bb13045f4b91a203db8c577b5" candidate_uuid: "dd43071d2a3745fdadd5812f273d476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8"
I20260808 02:07:38.461278 7592 raft_consensus.cc:3037] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.461364 7758 raft_consensus.cc:3037] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:38.462337 7758 raft_consensus.cc:2454] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd43071d2a3745fdadd5812f273d476c in term 1.
I20260808 02:07:38.462632 7862 leader_election.cc:302] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06114dcc94f54a5bbe4d2ef9c93981c8, dd43071d2a3745fdadd5812f273d476c; no voters:
I20260808 02:07:38.462819 8047 raft_consensus.cc:2781] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:38.462867 7592 raft_consensus.cc:2454] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd43071d2a3745fdadd5812f273d476c in term 1.
I20260808 02:07:38.463059 8047 raft_consensus.cc:686] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [term 1 LEADER]: Becoming Leader. State: Replica: dd43071d2a3745fdadd5812f273d476c, State: Running, Role: LEADER
I20260808 02:07:38.463361 8047 consensus_queue.cc:237] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } }
I20260808 02:07:38.464807 8089 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: TxnStatusTablet state changed. Reason: New leader dd43071d2a3745fdadd5812f273d476c. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.464928 8089 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:38.465423 8094 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:07:38.465612 8094 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:07:38.466079 7374 catalog_manager.cc:5495] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c reported cstate change: term changed from 0 to 1, leader changed from <none> to dd43071d2a3745fdadd5812f273d476c (127.5.209.67). New cstate: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } health_report { overall_health: HEALTHY } } }
I20260808 02:07:38.508250 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.504013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.5.209.67:33807)
0808 02:07:38.504159 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.504162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.504212 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:07:38.504433 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.504436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.504508 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.504595 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:38.504905 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.505587 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.506814 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.507026 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.507124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.507449 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.507878 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.507964 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.508079 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.508137 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:07:38.508306 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.503889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33807 (local address 127.0.0.1:48114)
0808 02:07:38.504151 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.504166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.504283 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.504642 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.504645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.504654 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:38.504792 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.504799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.505694 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.505698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.506695 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.506706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.507135 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.507139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.507241 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.507301 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.507972 (+ 671us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.507988 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.508068 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.508164 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260808 02:07:38.511832 7592 raft_consensus.cc:1261] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Refusing update from remote peer dd43071d2a3745fdadd5812f273d476c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.511916 7758 raft_consensus.cc:1261] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Refusing update from remote peer dd43071d2a3745fdadd5812f273d476c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.512377 8047 consensus_queue.cc:1035] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.512568 8089 consensus_queue.cc:1035] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.514312 8050 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: TxnStatusTablet state changed. Reason: New leader dd43071d2a3745fdadd5812f273d476c. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.514431 8050 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.514380 8075 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: TxnStatusTablet state changed. Reason: New leader dd43071d2a3745fdadd5812f273d476c. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.514505 8075 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.516225 8047 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.516342 8047 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:38.516870 8050 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.516978 8050 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 06114dcc94f54a5bbe4d2ef9c93981c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.517061 8053 mvcc.cc:205] Tried to move back new op lower bound from 7316090300458717184 to 7316090300268634112. Current Snapshot: MvccSnapshot[applied={T|T < 7316090300458717184 or (T in {7316090300458717184})}]
I20260808 02:07:38.519281 8047 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.519387 8047 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P dd43071d2a3745fdadd5812f273d476c: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:38.520074 8075 tablet_replica.cc:439] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd43071d2a3745fdadd5812f273d476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c47a923bad0441aae43ef7056fa4718" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46865 } } peers { permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 } } peers { permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 } } }
I20260808 02:07:38.520208 8075 tablet_replica.cc:442] T ff3e6e3bb13045f4b91a203db8c577b5 P 7c47a923bad0441aae43ef7056fa4718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:38.526789 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.523009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.5.209.65:46865)
0808 02:07:38.523122 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.523127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.523226 (+ 99us) server_negotiation.cc:407] Connection header received
0808 02:07:38.523417 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.523420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.523486 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.523590 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:38.523853 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.524379 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.525243 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.525560 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.525681 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.525958 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.526402 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.526534 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.526598 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.526660 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:07:38.537379 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.533893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:55885 (local address 127.5.209.67:33807)
0808 02:07:38.534034 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.534039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.534265 (+ 226us) server_negotiation.cc:407] Connection header received
0808 02:07:38.534420 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.534423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.534479 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.534561 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:38.534829 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.535316 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.536274 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.536495 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.536598 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.536927 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.536992 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.537073 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.537170 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.537219 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:07:38.537426 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.533798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33807 (local address 127.5.209.65:55885)
0808 02:07:38.534183 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.534194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.534310 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.534579 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.534582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.534589 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:38.534722 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.534727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.535441 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.535444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.536149 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.536158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.536611 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.536614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.536724 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.536767 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.537079 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.537087 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.537160 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.537233 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20260808 02:07:38.544548 7947 raft_consensus.cc:1261] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Refusing update from remote peer 7c47a923bad0441aae43ef7056fa4718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.544687 7758 raft_consensus.cc:1261] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Refusing update from remote peer 7c47a923bad0441aae43ef7056fa4718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:38.545045 8045 consensus_queue.cc:1035] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd43071d2a3745fdadd5812f273d476c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.545297 8075 consensus_queue.cc:1035] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06114dcc94f54a5bbe4d2ef9c93981c8" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:38.571058 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.565866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46865 (local address 127.5.209.67:48189)
0808 02:07:38.566938 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:38.566955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:38.567067 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:38.567473 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:38.567476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:38.567485 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:38.567709 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.567717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.568560 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.568564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.569584 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:38.569596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.570263 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.570268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:38.570368 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.570434 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.570761 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.570770 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:07:38.570832 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 02:07:38.570917 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":158,"threads_started":1}
I20260808 02:07:38.571045 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.565976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:48189 (local address 127.5.209.65:46865)
0808 02:07:38.566099 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.566104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.567026 (+ 922us) server_negotiation.cc:407] Connection header received
0808 02:07:38.567166 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.567170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.567231 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.567323 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:38.567872 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.568387 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.569737 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.570029 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.570155 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.570584 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:38.570675 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:38.570757 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.570851 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.570900 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:07:38.602161 7380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:38.602473 7380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:38.605645 7380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8a3e553b64844f5acb7fc0433235aea: Sending DeleteTablet for 3 replicas of tablet b641ebee336a4005ac9a37cc57093406
I20260808 02:07:38.606333 7564 tablet_service.cc:1514] Processing DeleteTablet for tablet b641ebee336a4005ac9a37cc57093406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:41528
I20260808 02:07:38.606598 8064 tablet_replica.cc:330] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: stopping tablet replica
I20260808 02:07:38.606765 8064 raft_consensus.cc:2229] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:38.607062 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet b641ebee336a4005ac9a37cc57093406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:50482
I20260808 02:07:38.607187 8064 raft_consensus.cc:2258] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.607344 7922 tablet_service.cc:1514] Processing DeleteTablet for tablet b641ebee336a4005ac9a37cc57093406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:38 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48102
I20260808 02:07:38.607460 8066 tablet_replica.cc:330] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: stopping tablet replica
I20260808 02:07:38.607568 8065 tablet_replica.cc:330] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: stopping tablet replica
I20260808 02:07:38.607614 8066 raft_consensus.cc:2229] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:38.607688 8065 raft_consensus.cc:2229] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:38.607733 8066 raft_consensus.cc:2258] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.607806 8065 raft_consensus.cc:2258] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:38.608610 8065 ts_tablet_manager.cc:1905] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.608788 8064 ts_tablet_manager.cc:1905] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.612063 8064 ts_tablet_manager.cc:1918] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:38.612063 8065 ts_tablet_manager.cc:1918] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:38.612149 8065 log.cc:1196] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/b641ebee336a4005ac9a37cc57093406
I20260808 02:07:38.612149 8064 log.cc:1196] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/b641ebee336a4005ac9a37cc57093406
I20260808 02:07:38.612426 8065 ts_tablet_manager.cc:1939] T b641ebee336a4005ac9a37cc57093406 P dd43071d2a3745fdadd5812f273d476c: Deleting consensus metadata
I20260808 02:07:38.612426 8064 ts_tablet_manager.cc:1939] T b641ebee336a4005ac9a37cc57093406 P 7c47a923bad0441aae43ef7056fa4718: Deleting consensus metadata
I20260808 02:07:38.613142 7361 catalog_manager.cc:4841] TS dd43071d2a3745fdadd5812f273d476c (127.5.209.67:33807): tablet b641ebee336a4005ac9a37cc57093406 (table test-table [id=dc30477cbd9e40c09391ece0c714ac55]) successfully deleted
I20260808 02:07:38.613142 7358 catalog_manager.cc:4841] TS 7c47a923bad0441aae43ef7056fa4718 (127.5.209.65:46865): tablet b641ebee336a4005ac9a37cc57093406 (table test-table [id=dc30477cbd9e40c09391ece0c714ac55]) successfully deleted
I20260808 02:07:38.613902 8066 ts_tablet_manager.cc:1905] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:38.617250 8066 ts_tablet_manager.cc:1918] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:38.617410 8066 log.cc:1196] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/b641ebee336a4005ac9a37cc57093406
I20260808 02:07:38.617748 8066 ts_tablet_manager.cc:1939] T b641ebee336a4005ac9a37cc57093406 P 06114dcc94f54a5bbe4d2ef9c93981c8: Deleting consensus metadata
I20260808 02:07:38.618480 7360 catalog_manager.cc:4841] TS 06114dcc94f54a5bbe4d2ef9c93981c8 (127.5.209.66:35479): tablet b641ebee336a4005ac9a37cc57093406 (table test-table [id=dc30477cbd9e40c09391ece0c714ac55]) successfully deleted
Aug 08 02:07:38 dist-test-slave-bw88 krb5kdc[7320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:38.626505 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.613905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.5.209.65:46865)
0808 02:07:38.614080 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.614086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.614786 (+ 700us) server_negotiation.cc:407] Connection header received
0808 02:07:38.614997 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.615000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.615047 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.615098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:38.616304 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.617058 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.618501 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.618814 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.618880 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.622486 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:38.622503 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:38.622516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:38.622540 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:38.624517 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.624982 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.624987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.624989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.625073 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.625388 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.625392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.625394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.625651 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:38.625883 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.626117 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.626381 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 02:07:38.627063 7634 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532
I20260808 02:07:38.637679 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:38.628779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.5.209.126:33629)
0808 02:07:38.628921 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:38.628926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:38.629323 (+ 397us) server_negotiation.cc:407] Connection header received
0808 02:07:38.629494 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:38.629497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:38.629548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:38.629640 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:38.630466 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.631319 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.632247 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:38.632487 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:38.632601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:38.634113 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:38.634117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:38.634120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:38.634147 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:38.635619 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.636183 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.636194 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.636196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.636274 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:38.636558 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:38.636562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:38.636564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:38.636754 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:38.636957 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:38.637330 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 02:07:38.637488 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 02:07:38.638075 7380 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58496
I20260808 02:07:38.639611 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 7478
I20260808 02:07:38.657601 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 7655
I20260808 02:07:38.671311 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 7832
I20260808 02:07:38.684537 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 7338
2026-08-08T02:07:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3058 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:38 dist-test-slave-bw88 krb5kdc[8121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:38 dist-test-slave-bw88 krb5kdc[8121](info): set up 2 sockets
Aug 08 02:07:38 dist-test-slave-bw88 krb5kdc[8121](info): commencing operation
krb5kdc: starting...
W20260808 02:07:39.873010 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:39 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:39.996903 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:45071
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:40.110858 8135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:40.111142 8135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:40.111222 8135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:40.115343 8135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:40.115414 8135 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:07:40.115455 8135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:40.115511 8135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:40.115549 8135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:40.115584 8135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:40.121071 8135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:45071
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:40.122215 8135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:40.123241 8135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:40.131944 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.132126 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.131953 8148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:40.132246 8135 server_base.cc:1034] running on GCE node
I20260808 02:07:40.132860 8135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:40.134061 8135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:40.135226 8135 hybrid_clock.cc:648] HybridClock initialized: now 1786154860135200 us; error 37 us; skew 500 ppm
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:40.138010 8135 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:40.138867 8135 webserver.cc:460] Webserver started at http://127.5.209.126:46005/ using document root <none> and password file <none>
I20260808 02:07:40.139402 8135 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:40.139461 8135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:40.139683 8135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:40.141734 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "fe6aaf3f5ddf407ebf5566def14e61aa"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "2eed6ddc28d3baaf6d2fe438f59fbafa"
server_key_iv: "d2721e2f39ac3b50122627b625c4610a"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.142261 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "fe6aaf3f5ddf407ebf5566def14e61aa"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "2eed6ddc28d3baaf6d2fe438f59fbafa"
server_key_iv: "d2721e2f39ac3b50122627b625c4610a"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.145887 8135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:40.148208 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.149523 8135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:40.149655 8135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "fe6aaf3f5ddf407ebf5566def14e61aa"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "2eed6ddc28d3baaf6d2fe438f59fbafa"
server_key_iv: "d2721e2f39ac3b50122627b625c4610a"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.149770 8135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:40.163751 8135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:40.166675 8135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:40.166884 8135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:40.175024 8135 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:45071
I20260808 02:07:40.175027 8237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:45071 every 8 connection(s)
I20260808 02:07:40.175956 8135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:40.178771 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:40.182890 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 8135
I20260808 02:07:40.183017 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:40.183288 5957 external_mini_cluster.cc:1428] Setting key 04c747f602f990854705ce12dfb590d0
I20260808 02:07:40.184780 8239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Bootstrap starting.
I20260808 02:07:40.187274 8239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:40.188063 8239 log.cc:824] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:40.189940 8239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: No bootstrap required, opened a new log
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:40.193276 8239 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } }
I20260808 02:07:40.193580 8239 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:40.193662 8239 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6aaf3f5ddf407ebf5566def14e61aa, State: Initialized, Role: FOLLOWER
I20260808 02:07:40.194180 8239 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } }
I20260808 02:07:40.194350 8239 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:40.194422 8239 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:40.194517 8239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:40.195587 8239 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } }
I20260808 02:07:40.195966 8239 leader_election.cc:302] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6aaf3f5ddf407ebf5566def14e61aa; no voters:
I20260808 02:07:40.196316 8239 leader_election.cc:288] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:40.196425 8249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:40.196686 8249 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [term 1 LEADER]: Becoming Leader. State: Replica: fe6aaf3f5ddf407ebf5566def14e61aa, State: Running, Role: LEADER
I20260808 02:07:40.196996 8249 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } }
I20260808 02:07:40.197700 8239 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:40.198520 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } } }
I20260808 02:07:40.198669 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:40.198891 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.184807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.5.209.126:45071)
0808 02:07:40.185294 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.185313 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.185403 (+ 90us) server_negotiation.cc:407] Connection header received
0808 02:07:40.186046 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.186063 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.186381 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.186656 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.187596 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.188686 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.189622 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.190139 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.190302 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.192772 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.192778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.192790 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.192817 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.195003 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.195445 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.195454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.195464 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.195623 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.195918 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.195921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.195923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.196496 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.197416 (+ 920us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.197568 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.197879 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":176,"threads_started":1}
I20260808 02:07:40.199047 8252 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6aaf3f5ddf407ebf5566def14e61aa. Latest consensus state: current_term: 1 leader_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6aaf3f5ddf407ebf5566def14e61aa" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45071 } } }
I20260808 02:07:40.199134 8252 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:40.199671 8263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:40.202926 8263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:40.208442 8263 catalog_manager.cc:1347] Generated new cluster ID: 2e75724968664486b8be4299b650ed96
I20260808 02:07:40.208513 8263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:40.240197 8263 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:40.241328 8263 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:40.247921 8263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Generated new TSK 0
I20260808 02:07:40.248600 8263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:40.281231 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45071
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:40.413002 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:40.413251 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:40.413331 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:40.417436 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:40.417490 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:40.417562 8279 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:07:40.417589 8279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:40.417632 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:40.422852 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:40.423910 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:40.424934 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:40.434536 8290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.434594 8287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.434592 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:40.434844 8279 server_base.cc:1034] running on GCE node
I20260808 02:07:40.435225 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:40.435807 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:40.436964 8279 hybrid_clock.cc:648] HybridClock initialized: now 1786154860436933 us; error 43 us; skew 500 ppm
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:40.443017 8279 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:40.443856 8279 webserver.cc:460] Webserver started at http://127.5.209.65:36223/ using document root <none> and password file <none>
I20260808 02:07:40.444403 8279 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:40.444487 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:40.444721 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:40.446805 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "951a0b03757b4f27876054fa1457f8f7"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "ddfd7f3b91294190af1d3ca41f30e194"
server_key_iv: "730e22631311fcf86c6b9f86a6a4141c"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.447371 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "951a0b03757b4f27876054fa1457f8f7"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "ddfd7f3b91294190af1d3ca41f30e194"
server_key_iv: "730e22631311fcf86c6b9f86a6a4141c"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.451069 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 02:07:40.453617 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.454854 8279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:07:40.454979 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "951a0b03757b4f27876054fa1457f8f7"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "ddfd7f3b91294190af1d3ca41f30e194"
server_key_iv: "730e22631311fcf86c6b9f86a6a4141c"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.455082 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:40.468847 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:40.471786 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:40.471977 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:40.473426 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:40.473491 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.473562 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:40.473613 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:40.484458 8279 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:46591
I20260808 02:07:40.484740 8449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:46591 every 8 connection(s)
I20260808 02:07:40.485526 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:40.488204 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.474286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:43803 (local address 127.5.209.126:45071)
0808 02:07:40.474427 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.474430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.475447 (+ 1017us) server_negotiation.cc:407] Connection header received
0808 02:07:40.476468 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.476475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.476545 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.476657 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.478050 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.478962 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.481111 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.481584 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.481697 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.484636 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.484642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.484646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.484678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.486402 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.486890 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.486893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.486894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.486971 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.487260 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.487262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.487263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.487434 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.487510 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.487967 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.488072 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:07:40.488299 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 8279
I20260808 02:07:40.488416 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:40.488722 5957 external_mini_cluster.cc:1428] Setting key f7d75511bb036bba8537168e351acbbe
I20260808 02:07:40.489224 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.474818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.65:43803)
0808 02:07:40.475307 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.475347 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.476220 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.476810 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.476818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.477241 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.477891 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.477909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.479363 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.479369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.480948 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.480962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.481753 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.481756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.481894 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.482496 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.482522 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.484431 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.486541 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.486545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.486556 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.486801 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.487052 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.487055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.487056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.487184 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.487556 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.487561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.487773 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.487920 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.488238 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":146,"threads_started":1}
I20260808 02:07:40.490635 8450 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45071
I20260808 02:07:40.490769 8450 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:40.491211 8450 heartbeater.cc:511] Master 127.5.209.126:45071 requested a full tablet report, sending...
I20260808 02:07:40.492650 8181 ts_manager.cc:194] Registered new tserver with Master: 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591)
I20260808 02:07:40.493892 8181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:43803
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:40.501605 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.493213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.65:49995)
0808 02:07:40.493613 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.493630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.493765 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.494240 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.494243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.494446 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.494546 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.494551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.495601 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.495605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.496230 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.496239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.496911 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.496914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.497001 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.497614 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.497626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.497973 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.500057 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.500059 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.500061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.500254 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.500549 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.500551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.500552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.500628 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.501081 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.501083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.501171 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.501253 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.501416 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260808 02:07:40.501688 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.493292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:49995 (local address 127.5.209.126:45071)
0808 02:07:40.493436 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.493442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.493694 (+ 252us) server_negotiation.cc:407] Connection header received
0808 02:07:40.493873 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.493878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.493928 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.493982 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.494666 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.495451 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.496354 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.496769 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.496884 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.498129 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.498135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.498141 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.498174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.499951 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.500353 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.500357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.500359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.500451 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.500717 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.500721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.500722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.500981 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.501094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.501353 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.501483 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:40.521247 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45071
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:40.639479 8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:40.639732 8461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:40.639782 8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:40.643893 8461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:40.643954 8461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:40.643994 8461 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:07:40.644048 8461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:40.644093 8461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:40.649681 8461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:40.650768 8461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:40.651850 8461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:40.662936 8468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.662930 8469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:40.663268 8461 server_base.cc:1034] running on GCE node
W20260808 02:07:40.662932 8471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:40.663906 8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:40.664598 8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:40.665760 8461 hybrid_clock.cc:648] HybridClock initialized: now 1786154860665744 us; error 26 us; skew 500 ppm
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:40.668598 8461 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:40.669663 8461 webserver.cc:460] Webserver started at http://127.5.209.66:34901/ using document root <none> and password file <none>
I20260808 02:07:40.670274 8461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:40.670367 8461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:40.670612 8461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:40.672816 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "9bd8ee07e76443e188e1bd23b50f7016"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "77bbd893132b85f5ee7d8cefdd47cbd2"
server_key_iv: "b3668a624d234374c9973a9f74702aa3"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.673475 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "9bd8ee07e76443e188e1bd23b50f7016"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "77bbd893132b85f5ee7d8cefdd47cbd2"
server_key_iv: "b3668a624d234374c9973a9f74702aa3"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.677241 8461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:07:40.679774 8479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.681118 8461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:40.681227 8461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "9bd8ee07e76443e188e1bd23b50f7016"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "77bbd893132b85f5ee7d8cefdd47cbd2"
server_key_iv: "b3668a624d234374c9973a9f74702aa3"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.681310 8461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:40.693224 8461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:40.696206 8461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:40.696370 8461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:40.698127 8461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:40.698197 8461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.698257 8461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:40.698285 8461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:40.710125 8461 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:45751
I20260808 02:07:40.710196 8628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:45751 every 8 connection(s)
I20260808 02:07:40.711076 8461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:40.712769 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.698820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:48325 (local address 127.5.209.126:45071)
0808 02:07:40.698982 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.698988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.700519 (+ 1531us) server_negotiation.cc:407] Connection header received
0808 02:07:40.702159 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.702165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.702232 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.702307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.704073 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.704623 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.705968 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.706208 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.706324 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.708875 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.708878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.708880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.708923 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.710588 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.711096 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.711100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.711102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.711209 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.711591 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.711596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.711597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.711853 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.711959 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.712522 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.712638 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:07:40.713590 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.699796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.66:48325)
0808 02:07:40.700348 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.700411 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.701863 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.702503 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.702516 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.703081 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.703872 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.703898 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.704755 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.704759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.705841 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.705852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.706382 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.706387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.706545 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.707187 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.707203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.708704 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.710712 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.710718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.710733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.710994 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.711306 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.711309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.711311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.711490 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.711975 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.711981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.712262 (+ 281us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.712420 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.712668 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":155,"threads_started":1}
I20260808 02:07:40.714778 8629 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45071
I20260808 02:07:40.714999 8629 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:40.715498 8629 heartbeater.cc:511] Master 127.5.209.126:45071 requested a full tablet report, sending...
I20260808 02:07:40.716564 8180 ts_manager.cc:194] Registered new tserver with Master: 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751)
I20260808 02:07:40.717378 8180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:48325
I20260808 02:07:40.718722 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 8461
I20260808 02:07:40.718852 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:40.719128 5957 external_mini_cluster.cc:1428] Setting key 5d91f2b93901afdfc457a6c5f76de1f8
I20260808 02:07:40.725431 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.717434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:34403 (local address 127.5.209.126:45071)
0808 02:07:40.717688 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.717691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.717906 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:07:40.718035 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.718037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.718076 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.718140 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.718691 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.719417 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.720518 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.720764 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.720846 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.722225 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.722228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.722230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.722252 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.723584 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.724129 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.724132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.724133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.724227 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.724541 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.724543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.724545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.724752 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.724828 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.725172 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.725262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260808 02:07:40.725632 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.717306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.66:34403)
0808 02:07:40.717821 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.717833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.717940 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.718183 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.718186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.718428 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.718545 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.718552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.719590 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.719593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.720428 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.720439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.720881 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.720885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.721007 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.721564 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.721585 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.722090 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.723715 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.723720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.723723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.724006 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.724320 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.724324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.724326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.724439 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.724852 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.724858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.724997 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.725121 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.725295 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:40.748378 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45071
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:40.865533 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:40.865844 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:40.865937 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:40.870043 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:40.870123 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:40.870193 8641 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:07:40.870232 8641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:07:40.870302 8641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:40.875994 8641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:40.877143 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:40.878258 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:40.888491 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.888563 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:40.888494 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:40.888556 8641 server_base.cc:1034] running on GCE node
I20260808 02:07:40.889163 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:40.889850 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:40.891036 8641 hybrid_clock.cc:648] HybridClock initialized: now 1786154860891010 us; error 35 us; skew 500 ppm
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:40.893990 8641 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:40.894958 8641 webserver.cc:460] Webserver started at http://127.5.209.67:39297/ using document root <none> and password file <none>
I20260808 02:07:40.895545 8641 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:40.895669 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:40.895905 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:40.898129 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "502ee16253df4284ac28e9e4ff8df4f8"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "22ea4ec9cb10d3f24e229e2fea3955c3"
server_key_iv: "7d0c31a8b1a54baa87ffb7d1c8719a99"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.898741 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "502ee16253df4284ac28e9e4ff8df4f8"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "22ea4ec9cb10d3f24e229e2fea3955c3"
server_key_iv: "7d0c31a8b1a54baa87ffb7d1c8719a99"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.902519 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:07:40.905055 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.906318 8641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:40.906450 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "502ee16253df4284ac28e9e4ff8df4f8"
format_stamp: "Formatted at 2026-08-08 02:07:40 on dist-test-slave-bw88"
server_key: "22ea4ec9cb10d3f24e229e2fea3955c3"
server_key_iv: "7d0c31a8b1a54baa87ffb7d1c8719a99"
server_key_version: "encryptionkey@0"
I20260808 02:07:40.906589 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:40.916759 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:40.919641 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:40.919831 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:40.921322 8641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:40.921460 8641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:40.921532 8641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:40.921586 8641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:40.932842 8641 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:41067
I20260808 02:07:40.933979 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:40.934207 8802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:41067 every 8 connection(s)
I20260808 02:07:40.935202 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 8641
I20260808 02:07:40.935336 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:40.935672 5957 external_mini_cluster.cc:1428] Setting key 08c064e3e13af9d86408b405c0137fe9
I20260808 02:07:40.935796 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.921997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:46669 (local address 127.5.209.126:45071)
0808 02:07:40.922151 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.922156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.923527 (+ 1371us) server_negotiation.cc:407] Connection header received
0808 02:07:40.925019 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.925025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.925087 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.925201 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.926508 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.927286 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.928706 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.928940 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.929061 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.931581 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.931585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.931588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.931627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.933291 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.933955 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.933958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.933959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.934055 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.934455 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.934460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.934462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.934701 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.934818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.935485 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.935628 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:07:40.936717 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.922870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.67:46669)
0808 02:07:40.923355 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.923411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.924683 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.925429 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.925440 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.925897 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.926354 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.926368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.927402 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.927407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.928575 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.928586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.929121 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.929125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.929359 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.929979 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.929995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.931416 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.933502 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.933508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.933521 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.933851 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.934175 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.934178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.934179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.934355 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.934827 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.934836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.935177 (+ 341us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.935403 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.935746 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260808 02:07:40.938056 8803 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45071
I20260808 02:07:40.938290 8803 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:40.938889 8803 heartbeater.cc:511] Master 127.5.209.126:45071 requested a full tablet report, sending...
I20260808 02:07:40.940042 8181 ts_manager.cc:194] Registered new tserver with Master: 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:40.940748 8181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:46669
I20260808 02:07:40.947613 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.940304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.5.209.67:42373)
0808 02:07:40.941112 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.941125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.941246 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.941546 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.941549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.941785 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:40.941897 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.941905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.942622 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.942625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.943264 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.943273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.943676 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.943678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.943749 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.944202 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:40.944213 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:40.944540 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:40.946226 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.946229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.946230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.946423 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.946738 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:40.946740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:40.946741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:40.946815 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:40.947204 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:40.947206 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:40.947280 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 02:07:40.947360 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:07:40.947465 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":96,"threads_started":1}
I20260808 02:07:40.947710 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.940362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:42373 (local address 127.5.209.126:45071)
0808 02:07:40.940487 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.940492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.941206 (+ 714us) server_negotiation.cc:407] Connection header received
0808 02:07:40.941389 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.941392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.941442 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.941505 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.942007 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.942488 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.943364 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.943584 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.943660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.944663 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.944667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.944669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.944691 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.946137 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.946534 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.946537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.946538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.946640 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.946917 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.946920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.946922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.947115 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.947205 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.947464 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.947562 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:07:40.950007 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:07:40 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:40.978538 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.969978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.5.209.126:45071)
0808 02:07:40.970143 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.970146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.970160 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:07:40.970203 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.970205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.970245 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.970332 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:40.971086 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.971623 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.972493 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.972710 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.972825 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.975361 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:40.975365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:40.975367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:40.975393 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:40.976785 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.977241 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.977245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.977247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.977325 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:40.977657 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:40.977661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:40.977663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:40.977854 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:40.977992 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:40.978231 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:07:40.978355 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:07:40.981616 8181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:40.984731 8181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:41.000355 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.995066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.5.209.66:45751)
0808 02:07:40.995541 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.995553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.995578 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:07:40.995660 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.995666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.995881 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.996165 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:40.996287 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997460 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999094 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.999650 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999782 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.999960 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.000093 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.000196 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":133,"threads_started":1}
I20260808 02:07:41.000344 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.993810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:45751 (local address 127.0.0.1:57418)
0808 02:07:40.994221 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.994275 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.994412 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.996049 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.996052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.996070 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:40.996208 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.996215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997597 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.997601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.998934 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.998945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.999782 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.999899 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:40.999940 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.000062 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.000189 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":176,"thread_start_us":75,"threads_started":1}
I20260808 02:07:41.000648 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.994598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46591 (local address 127.0.0.1:33316)
0808 02:07:40.995442 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.995456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.995545 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.996012 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.996015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.996031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:40.996336 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.996342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997746 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.997752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.998861 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.998873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.000187 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.000193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.000297 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.000321 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.000394 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.000514 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":48,"thread_start_us":97,"threads_started":1}
I20260808 02:07:41.000749 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.994441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.5.209.67:41067)
0808 02:07:40.994816 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.994829 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.995199 (+ 370us) server_negotiation.cc:407] Connection header received
0808 02:07:40.995358 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.995364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.995663 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.995837 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:40.996155 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997034 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999012 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.999829 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.000008 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.000169 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.000500 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.000598 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":175,"threads_started":1}
I20260808 02:07:41.001104 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.994774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.5.209.65:46591)
0808 02:07:40.995056 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:40.995064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:40.995640 (+ 576us) server_negotiation.cc:407] Connection header received
0808 02:07:40.995743 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:40.995747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:40.995895 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:40.996045 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:40.996446 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997591 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999026 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.999829 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.999958 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.000115 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.000836 (+ 721us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.000938 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260808 02:07:41.001358 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:40.994238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:41067 (local address 127.0.0.1:34780)
0808 02:07:40.995084 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:40.995101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:40.995222 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:40.995824 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:40.995830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:40.995854 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:40.996012 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.996020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:40.997238 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:40.997241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:40.998855 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:40.998867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.000212 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.000215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.000306 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.000322 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.000394 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.001204 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":516,"thread_start_us":106,"threads_started":1}
I20260808 02:07:41.003633 8542 tablet_service.cc:1467] Processing CreateTablet for tablet bebd5ef2adc14792ac0d497f700ef86f (DEFAULT_TABLE table=test-table [id=73fdaf1184f8494fb2ec94282506705a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.004209 8717 tablet_service.cc:1467] Processing CreateTablet for tablet bebd5ef2adc14792ac0d497f700ef86f (DEFAULT_TABLE table=test-table [id=73fdaf1184f8494fb2ec94282506705a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.004125 8340 tablet_service.cc:1467] Processing CreateTablet for tablet bebd5ef2adc14792ac0d497f700ef86f (DEFAULT_TABLE table=test-table [id=73fdaf1184f8494fb2ec94282506705a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.004675 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebd5ef2adc14792ac0d497f700ef86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.005203 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebd5ef2adc14792ac0d497f700ef86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.005211 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebd5ef2adc14792ac0d497f700ef86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.011922 8830 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Bootstrap starting.
I20260808 02:07:41.013510 8831 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Bootstrap starting.
I20260808 02:07:41.013582 8832 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Bootstrap starting.
I20260808 02:07:41.014282 8830 tablet_bootstrap.cc:654] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.015379 8830 log.cc:824] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:41.015798 8832 tablet_bootstrap.cc:654] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.015798 8831 tablet_bootstrap.cc:654] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.016669 8831 log.cc:824] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:41.016669 8832 log.cc:824] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:41.017514 8830 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: No bootstrap required, opened a new log
I20260808 02:07:41.017726 8830 ts_tablet_manager.cc:1397] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:07:41.018522 8831 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: No bootstrap required, opened a new log
I20260808 02:07:41.018723 8831 ts_tablet_manager.cc:1397] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 02:07:41.018733 8832 tablet_bootstrap.cc:492] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: No bootstrap required, opened a new log
I20260808 02:07:41.018900 8832 ts_tablet_manager.cc:1397] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 02:07:41.021829 8830 raft_consensus.cc:348] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.022181 8830 raft_consensus.cc:374] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.022253 8830 raft_consensus.cc:729] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.022126 8831 raft_consensus.cc:348] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.022404 8831 raft_consensus.cc:374] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.022486 8831 raft_consensus.cc:729] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.022782 8830 consensus_queue.cc:260] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.022907 8831 consensus_queue.cc:260] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.023550 8629 heartbeater.cc:503] Master 127.5.209.126:45071 was elected leader, sending a full tablet report...
I20260808 02:07:41.023839 8830 ts_tablet_manager.cc:1428] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:41.023762 8832 raft_consensus.cc:348] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.024008 8832 raft_consensus.cc:374] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.024070 8832 raft_consensus.cc:729] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ee16253df4284ac28e9e4ff8df4f8, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.024489 8832 consensus_queue.cc:260] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.024600 8831 ts_tablet_manager.cc:1428] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:41.024955 8450 heartbeater.cc:503] Master 127.5.209.126:45071 was elected leader, sending a full tablet report...
I20260808 02:07:41.025355 8803 heartbeater.cc:503] Master 127.5.209.126:45071 was elected leader, sending a full tablet report...
I20260808 02:07:41.026376 8832 ts_tablet_manager.cc:1428] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
W20260808 02:07:41.186184 8804 tablet.cc:2369] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:07:41.212508 8630 tablet.cc:2369] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:07:41.212508 8836 raft_consensus.cc:482] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:41.212770 8836 raft_consensus.cc:504] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.214205 8836 leader_election.cc:288] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.218273 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.214492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:41067 (local address 127.5.209.66:43469)
0808 02:07:41.214839 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.214852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.215001 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.215278 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.215281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.215322 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:41.215460 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.215465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.216275 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.216279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.217099 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.217110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.217823 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.217920 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.217954 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.218015 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.218069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":19,"spinlock_wait_cycles":896,"thread_start_us":63,"threads_started":1}
I20260808 02:07:41.218288 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.214598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:43469 (local address 127.5.209.67:41067)
0808 02:07:41.214735 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.214740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.214946 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:07:41.215112 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.215115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.215158 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.215251 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:41.215570 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.216172 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217227 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.217704 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217823 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.217883 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.218059 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.218148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:07:41.218309 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46591 (local address 127.5.209.66:48169)
0808 02:07:41.214763 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.214775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.214934 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.215232 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.215239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.215258 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:41.215448 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.215455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.216353 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.216356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.217133 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.217142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.217866 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.217984 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.218025 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.218108 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.218170 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":20,"spinlock_wait_cycles":16512,"thread_start_us":124,"threads_started":1}
I20260808 02:07:41.218359 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.214541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:48169 (local address 127.5.209.65:46591)
0808 02:07:41.214686 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.214691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.214854 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:07:41.215069 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.215072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.215135 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.215195 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:41.215620 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.216228 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217276 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.217729 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.217861 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.217915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.218145 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.218220 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:07:41.218952 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebd5ef2adc14792ac0d497f700ef86f" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8" is_pre_election: true
I20260808 02:07:41.219017 8384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebd5ef2adc14792ac0d497f700ef86f" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951a0b03757b4f27876054fa1457f8f7" is_pre_election: true
I20260808 02:07:41.219285 8745 raft_consensus.cc:2454] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 0.
I20260808 02:07:41.219350 8384 raft_consensus.cc:2454] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 0.
I20260808 02:07:41.219815 8484 leader_election.cc:302] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.220075 8836 raft_consensus.cc:2781] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:41.220149 8836 raft_consensus.cc:482] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:41.220198 8836 raft_consensus.cc:3037] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.221818 8836 raft_consensus.cc:504] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.222213 8836 leader_election.cc:288] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 election: Requested vote from peers 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.222556 8384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebd5ef2adc14792ac0d497f700ef86f" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951a0b03757b4f27876054fa1457f8f7"
I20260808 02:07:41.222568 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebd5ef2adc14792ac0d497f700ef86f" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8"
I20260808 02:07:41.222687 8745 raft_consensus.cc:3037] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.222687 8384 raft_consensus.cc:3037] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.224123 8384 raft_consensus.cc:2454] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 1.
I20260808 02:07:41.224478 8483 leader_election.cc:302] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.224548 8745 raft_consensus.cc:2454] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 1.
I20260808 02:07:41.224748 8836 raft_consensus.cc:2781] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:41.224994 8836 raft_consensus.cc:686] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Running, Role: LEADER
I20260808 02:07:41.225400 8836 consensus_queue.cc:237] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.228582 8179 catalog_manager.cc:5495] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } health_report { overall_health: UNKNOWN } } }
W20260808 02:07:41.237403 8451 tablet.cc:2369] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:07:41 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:41.263161 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.255037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45071 (local address 127.0.0.1:39374)
0808 02:07:41.255455 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.255502 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.255623 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.255848 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.255850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.256230 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:41.256329 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.256336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.257030 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.257033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.257652 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.257662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.258056 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.258058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.258120 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.258698 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:41.258709 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:41.260150 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:41.261816 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:41.261825 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:41.261830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:41.262057 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:41.262293 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:41.262295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:41.262297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:41.262373 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:41.262705 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:41.262710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:41.262816 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.262909 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.263030 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":110,"threads_started":1}
I20260808 02:07:41.263200 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.255193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.5.209.126:45071)
0808 02:07:41.255379 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.255383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.255590 (+ 207us) server_negotiation.cc:407] Connection header received
0808 02:07:41.255701 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.255704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.255743 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.255811 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:41.256430 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.256939 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.257758 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.257962 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.258039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.260273 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:41.260276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:41.260278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:41.260299 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:41.261710 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:41.262140 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:41.262143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:41.262144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:41.262215 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:41.262444 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:41.262447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:41.262448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:41.262599 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:41.262687 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.262964 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.263091 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":70}
I20260808 02:07:41.267117 8179 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:39374:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:07:41.267936 8179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:41.273047 8340 tablet_service.cc:1467] Processing CreateTablet for tablet d834ed8f32fb451fbfa4a56f97133cd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=314e25eea65e477e9df6b291025510b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:41.273118 8542 tablet_service.cc:1467] Processing CreateTablet for tablet d834ed8f32fb451fbfa4a56f97133cd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=314e25eea65e477e9df6b291025510b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:41.273216 8717 tablet_service.cc:1467] Processing CreateTablet for tablet d834ed8f32fb451fbfa4a56f97133cd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=314e25eea65e477e9df6b291025510b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:07:41.273864 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d834ed8f32fb451fbfa4a56f97133cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.273864 8542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d834ed8f32fb451fbfa4a56f97133cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.274004 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d834ed8f32fb451fbfa4a56f97133cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.276513 8832 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: Bootstrap starting.
I20260808 02:07:41.276823 8831 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: Bootstrap starting.
I20260808 02:07:41.277074 8830 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: Bootstrap starting.
I20260808 02:07:41.277570 8832 tablet_bootstrap.cc:654] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.277889 8831 tablet_bootstrap.cc:654] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.278184 8830 tablet_bootstrap.cc:654] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.279007 8832 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: No bootstrap required, opened a new log
I20260808 02:07:41.279104 8832 ts_tablet_manager.cc:1397] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:41.279511 8830 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: No bootstrap required, opened a new log
I20260808 02:07:41.279588 8830 ts_tablet_manager.cc:1397] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:41.279649 8831 tablet_bootstrap.cc:492] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: No bootstrap required, opened a new log
I20260808 02:07:41.279708 8831 ts_tablet_manager.cc:1397] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:41.279625 8832 raft_consensus.cc:348] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.279739 8832 raft_consensus.cc:374] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.279770 8832 raft_consensus.cc:729] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ee16253df4284ac28e9e4ff8df4f8, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.279899 8832 consensus_queue.cc:260] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.280055 8830 raft_consensus.cc:348] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.280171 8830 raft_consensus.cc:374] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.280200 8830 raft_consensus.cc:729] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.280197 8831 raft_consensus.cc:348] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.280324 8831 raft_consensus.cc:374] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.280321 8830 consensus_queue.cc:260] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.280368 8831 raft_consensus.cc:729] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.280501 8831 consensus_queue.cc:260] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.280586 8830 ts_tablet_manager.cc:1428] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:41.280710 8840 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.280567 8836 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.280794 8837 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.280866 8840 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.280928 8837 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.280929 8836 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.281639 8832 ts_tablet_manager.cc:1428] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:41.282105 8831 ts_tablet_manager.cc:1428] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:41.338943 8836 raft_consensus.cc:482] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:41.339139 8836 raft_consensus.cc:504] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.339603 8836 leader_election.cc:288] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.340175 8384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d834ed8f32fb451fbfa4a56f97133cd4" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951a0b03757b4f27876054fa1457f8f7" is_pre_election: true
I20260808 02:07:41.340163 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d834ed8f32fb451fbfa4a56f97133cd4" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8" is_pre_election: true
I20260808 02:07:41.340342 8384 raft_consensus.cc:2454] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 0.
I20260808 02:07:41.340368 8745 raft_consensus.cc:2454] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 0.
I20260808 02:07:41.340669 8484 leader_election.cc:302] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.340878 8836 raft_consensus.cc:2781] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:41.340946 8836 raft_consensus.cc:482] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:41.340996 8836 raft_consensus.cc:3037] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.342126 8836 raft_consensus.cc:504] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.342470 8836 leader_election.cc:288] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 election: Requested vote from peers 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.342881 8384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d834ed8f32fb451fbfa4a56f97133cd4" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951a0b03757b4f27876054fa1457f8f7"
I20260808 02:07:41.342881 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d834ed8f32fb451fbfa4a56f97133cd4" candidate_uuid: "9bd8ee07e76443e188e1bd23b50f7016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8"
I20260808 02:07:41.343012 8745 raft_consensus.cc:3037] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.343014 8384 raft_consensus.cc:3037] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.344182 8384 raft_consensus.cc:2454] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 1.
I20260808 02:07:41.344182 8745 raft_consensus.cc:2454] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd8ee07e76443e188e1bd23b50f7016 in term 1.
I20260808 02:07:41.344475 8484 leader_election.cc:302] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.344687 8836 raft_consensus.cc:2781] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:41.344762 8836 raft_consensus.cc:686] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Running, Role: LEADER
I20260808 02:07:41.344871 8836 consensus_queue.cc:237] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.345655 8843 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: TxnStatusTablet state changed. Reason: New leader 9bd8ee07e76443e188e1bd23b50f7016. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.345775 8843 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:41.346114 8856 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:07:41.346385 8856 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:07:41.346679 8181 catalog_manager.cc:5495] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:41.366346 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.362245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:45751 (local address 127.0.0.1:57422)
0808 02:07:41.362425 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.362444 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.362595 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.362922 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.362924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.362930 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:41.363025 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.363030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.364024 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.364027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.364640 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.364648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.365213 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.365216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.365284 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.365330 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.366096 (+ 766us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.366110 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.366159 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.366233 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:07:41.366370 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.362365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.5.209.66:45751)
0808 02:07:41.362494 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.362498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.362531 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:07:41.362729 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.362732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.362797 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.362894 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.363125 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.363911 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.364748 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.365091 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.365189 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.365496 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.365995 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.366101 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.366172 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.366232 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:07:41.371630 8745 raft_consensus.cc:1261] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Refusing update from remote peer 9bd8ee07e76443e188e1bd23b50f7016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.371701 8384 raft_consensus.cc:1261] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Refusing update from remote peer 9bd8ee07e76443e188e1bd23b50f7016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.372393 8843 consensus_queue.cc:1035] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.372596 8836 consensus_queue.cc:1035] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.376503 8837 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: TxnStatusTablet state changed. Reason: New leader 9bd8ee07e76443e188e1bd23b50f7016. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.376647 8837 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.377068 8840 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: TxnStatusTablet state changed. Reason: New leader 9bd8ee07e76443e188e1bd23b50f7016. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.377182 8840 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.379897 8843 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.380012 8843 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:41.380366 8843 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.380479 8843 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 9bd8ee07e76443e188e1bd23b50f7016: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:07:41.381013 8859 mvcc.cc:205] Tried to move back new op lower bound from 7316090312171413504 to 7316090312070049792. Current Snapshot: MvccSnapshot[applied={T|T < 7316090312171413504}]
I20260808 02:07:41.381309 8837 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.381433 8837 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 951a0b03757b4f27876054fa1457f8f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.382964 8840 tablet_replica.cc:439] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9bd8ee07e76443e188e1bd23b50f7016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } } }
I20260808 02:07:41.383076 8840 tablet_replica.cc:442] T d834ed8f32fb451fbfa4a56f97133cd4 P 502ee16253df4284ac28e9e4ff8df4f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:07:41.385967 8861 mvcc.cc:205] Tried to move back new op lower bound from 7316090312171413504 to 7316090312070049792. Current Snapshot: MvccSnapshot[applied={T|T < 7316090312171413504}]
I20260808 02:07:41.391716 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.388130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.5.209.66:45751)
0808 02:07:41.388241 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.388244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.388294 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:07:41.388421 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.388424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.388468 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.388538 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.388817 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.389295 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.390607 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.390819 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.390973 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.391277 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.391349 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.391436 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.391529 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.391581 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:07:41.403213 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.399661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:44019 (local address 127.5.209.66:45751)
0808 02:07:41.399801 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.399807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.400105 (+ 298us) server_negotiation.cc:407] Connection header received
0808 02:07:41.400257 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.400261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.400305 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.400383 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.400659 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.401172 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.402114 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.402322 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.402426 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.402736 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.402813 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.402890 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.402978 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.403052 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":47}
I20260808 02:07:41.403218 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.399551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:45751 (local address 127.5.209.66:44019)
0808 02:07:41.400025 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.400038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.400157 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.400391 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.400394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.400401 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:41.400554 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.400560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.401284 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.401288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.401973 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.401983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.402429 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.402433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.402535 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.402580 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.402895 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.402903 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.402970 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.403066 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":24,"thread_start_us":119,"threads_started":1}
I20260808 02:07:41.410593 8745 raft_consensus.cc:1261] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Refusing update from remote peer 9bd8ee07e76443e188e1bd23b50f7016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.410713 8384 raft_consensus.cc:1261] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Refusing update from remote peer 9bd8ee07e76443e188e1bd23b50f7016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.411058 8843 consensus_queue.cc:1035] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.411252 8836 consensus_queue.cc:1035] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.417599 8860 mvcc.cc:205] Tried to move back new op lower bound from 7316090312333422592 to 7316090311582195712. Current Snapshot: MvccSnapshot[applied={T|T < 7316090312333422592}]
I20260808 02:07:41.458230 8178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:41.458428 8178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:41.460933 8178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Sending DeleteTablet for 3 replicas of tablet bebd5ef2adc14792ac0d497f700ef86f
I20260808 02:07:41.461812 8340 tablet_service.cc:1514] Processing DeleteTablet for tablet bebd5ef2adc14792ac0d497f700ef86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:41 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260808 02:07:41.461885 8717 tablet_service.cc:1514] Processing DeleteTablet for tablet bebd5ef2adc14792ac0d497f700ef86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:41 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34780
I20260808 02:07:41.462306 8884 tablet_replica.cc:330] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: stopping tablet replica
I20260808 02:07:41.462332 8541 tablet_service.cc:1514] Processing DeleteTablet for tablet bebd5ef2adc14792ac0d497f700ef86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:41 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:57418
I20260808 02:07:41.462687 8884 raft_consensus.cc:2229] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:41.462863 8887 tablet_replica.cc:330] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: stopping tablet replica
I20260808 02:07:41.462922 8884 raft_consensus.cc:2258] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:41.463443 8885 tablet_replica.cc:330] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: stopping tablet replica
I20260808 02:07:41.463572 8178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:07:41.463742 8885 raft_consensus.cc:2229] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:41.463937 8885 raft_consensus.cc:2258] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:07:41.463991 8178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:41.464330 8887 raft_consensus.cc:2229] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:41.464759 8887 raft_consensus.cc:2258] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:41.465505 8885 ts_tablet_manager.cc:1905] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:41.466276 8887 ts_tablet_manager.cc:1905] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:41.468905 8885 ts_tablet_manager.cc:1918] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:41.469048 8885 log.cc:1196] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/bebd5ef2adc14792ac0d497f700ef86f
I20260808 02:07:41.469414 8884 ts_tablet_manager.cc:1905] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:41.469434 8885 ts_tablet_manager.cc:1939] T bebd5ef2adc14792ac0d497f700ef86f P 951a0b03757b4f27876054fa1457f8f7: Deleting consensus metadata
I20260808 02:07:41.469918 8717 tablet_service.cc:1467] Processing CreateTablet for tablet f11a225235b14f12b7804f742ebd055c (DEFAULT_TABLE table=test-table [id=3dd8aa5764e6474887973d1a67a18ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.470197 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a225235b14f12b7804f742ebd055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.472150 8832 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Bootstrap starting.
I20260808 02:07:41.472416 8884 ts_tablet_manager.cc:1918] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:41.472515 8884 log.cc:1196] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/bebd5ef2adc14792ac0d497f700ef86f
I20260808 02:07:41.472832 8884 ts_tablet_manager.cc:1939] T bebd5ef2adc14792ac0d497f700ef86f P 502ee16253df4284ac28e9e4ff8df4f8: Deleting consensus metadata
I20260808 02:07:41.472869 8162 catalog_manager.cc:4841] TS 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591): tablet bebd5ef2adc14792ac0d497f700ef86f (table test-table [id=73fdaf1184f8494fb2ec94282506705a]) successfully deleted
I20260808 02:07:41.473176 8832 tablet_bootstrap.cc:654] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.473985 8163 catalog_manager.cc:4841] TS 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067): tablet bebd5ef2adc14792ac0d497f700ef86f (table test-table [id=73fdaf1184f8494fb2ec94282506705a]) successfully deleted
I20260808 02:07:41.474352 8541 tablet_service.cc:1467] Processing CreateTablet for tablet f11a225235b14f12b7804f742ebd055c (DEFAULT_TABLE table=test-table [id=3dd8aa5764e6474887973d1a67a18ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.474637 8541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a225235b14f12b7804f742ebd055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.475086 8887 ts_tablet_manager.cc:1918] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:41.475188 8887 log.cc:1196] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/bebd5ef2adc14792ac0d497f700ef86f
I20260808 02:07:41.475526 8887 ts_tablet_manager.cc:1939] T bebd5ef2adc14792ac0d497f700ef86f P 9bd8ee07e76443e188e1bd23b50f7016: Deleting consensus metadata
I20260808 02:07:41.477082 8830 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Bootstrap starting.
I20260808 02:07:41.477463 8340 tablet_service.cc:1467] Processing CreateTablet for tablet f11a225235b14f12b7804f742ebd055c (DEFAULT_TABLE table=test-table [id=3dd8aa5764e6474887973d1a67a18ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:41.477723 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11a225235b14f12b7804f742ebd055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:41.477931 8830 tablet_bootstrap.cc:654] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.478613 8832 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: No bootstrap required, opened a new log
I20260808 02:07:41.478716 8832 ts_tablet_manager.cc:1397] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260808 02:07:41.479169 8832 raft_consensus.cc:348] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.479297 8832 raft_consensus.cc:374] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.479333 8832 raft_consensus.cc:729] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ee16253df4284ac28e9e4ff8df4f8, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.479488 8832 consensus_queue.cc:260] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.479854 8832 ts_tablet_manager.cc:1428] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:07:41.480046 8831 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Bootstrap starting.
I20260808 02:07:41.480988 8831 tablet_bootstrap.cc:654] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:41.481003 8830 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: No bootstrap required, opened a new log
I20260808 02:07:41.481084 8830 ts_tablet_manager.cc:1397] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 02:07:41.481568 8830 raft_consensus.cc:348] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.481695 8830 raft_consensus.cc:374] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.481732 8830 raft_consensus.cc:729] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.481829 8162 catalog_manager.cc:4841] TS 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751): tablet bebd5ef2adc14792ac0d497f700ef86f (table test-table [id=73fdaf1184f8494fb2ec94282506705a]) successfully deleted
I20260808 02:07:41.481856 8830 consensus_queue.cc:260] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.482136 8830 ts_tablet_manager.cc:1428] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:41.484062 8831 tablet_bootstrap.cc:492] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: No bootstrap required, opened a new log
I20260808 02:07:41.484161 8831 ts_tablet_manager.cc:1397] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 02:07:41.484740 8831 raft_consensus.cc:348] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.484910 8831 raft_consensus.cc:374] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:41.484963 8831 raft_consensus.cc:729] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Initialized, Role: FOLLOWER
I20260808 02:07:41.485116 8831 consensus_queue.cc:260] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.485471 8831 ts_tablet_manager.cc:1428] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 02:07:41.493098 8837 raft_consensus.cc:482] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:41.493212 8837 raft_consensus.cc:504] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.494307 8837 leader_election.cc:288] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.500191 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:47987 (local address 127.5.209.66:45751)
0808 02:07:41.495357 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.495361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.495374 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:07:41.495424 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.495430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.495478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.495555 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:41.496255 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.496796 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.498034 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.498691 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.498801 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.498874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.499972 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.500053 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:07:41.500747 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.494478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:45751 (local address 127.5.209.65:47987)
0808 02:07:41.494727 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.494743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.494836 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.495861 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.495864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.495883 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:41.496143 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.496155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.496912 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.496916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.497894 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.497905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.499703 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.499709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.499821 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.499848 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.500532 (+ 684us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.500608 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260808 02:07:41.502064 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a225235b14f12b7804f742ebd055c" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016" is_pre_election: true
I20260808 02:07:41.502332 8570 raft_consensus.cc:2454] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:41.502799 8302 leader_election.cc:302] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.503113 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.495308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:32849 (local address 127.5.209.67:41067)
0808 02:07:41.495411 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.495415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.498345 (+ 2930us) server_negotiation.cc:407] Connection header received
0808 02:07:41.498424 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.498427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.498477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.498546 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:41.498919 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.499627 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.501058 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.501725 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.501803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.501849 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.502891 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.502989 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:07:41.503232 8837 raft_consensus.cc:2781] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:41.503288 8837 raft_consensus.cc:482] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:41.503326 8837 raft_consensus.cc:3037] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.503484 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.494878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:41067 (local address 127.5.209.65:32849)
0808 02:07:41.498206 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.498221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.498333 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.498663 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.498666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.498686 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:41.498787 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.498793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.499785 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.499790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.500895 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.500907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.502641 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.502645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.502734 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.502756 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.503247 (+ 491us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.503335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3257,"thread_start_us":113,"threads_started":1}
I20260808 02:07:41.504443 8837 raft_consensus.cc:504] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.504799 8837 leader_election.cc:288] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Requested vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:41.505151 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a225235b14f12b7804f742ebd055c" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016"
I20260808 02:07:41.505164 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a225235b14f12b7804f742ebd055c" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8" is_pre_election: true
I20260808 02:07:41.505270 8570 raft_consensus.cc:3037] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.505286 8745 raft_consensus.cc:2454] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:41.505777 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11a225235b14f12b7804f742ebd055c" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8"
I20260808 02:07:41.505889 8744 raft_consensus.cc:3037] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:41.506563 8570 raft_consensus.cc:2454] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:41.506881 8302 leader_election.cc:302] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:41.507072 8837 raft_consensus.cc:2781] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:41.507187 8744 raft_consensus.cc:2454] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:41.507297 8837 raft_consensus.cc:686] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Running, Role: LEADER
I20260808 02:07:41.507565 8837 consensus_queue.cc:237] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:41.510092 8179 catalog_manager.cc:5495] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "951a0b03757b4f27876054fa1457f8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:41.527752 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.523829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.5.209.65:46591)
0808 02:07:41.523977 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.523983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.524004 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:07:41.524104 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.524109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.524183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.524345 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.524595 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.525318 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.526197 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.526545 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.526641 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.526864 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.527383 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.527515 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.527566 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.527619 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:07:41.538784 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.534811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:45751 (local address 127.5.209.65:49231)
0808 02:07:41.535184 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.535221 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.535325 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.535627 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.535630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.535639 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:41.535780 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.535787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.536451 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.536455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.537470 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.537482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.537981 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.537985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.538085 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.538140 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.538427 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.538434 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.538509 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.538602 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260808 02:07:41.538789 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.534896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:49231 (local address 127.5.209.66:45751)
0808 02:07:41.535029 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.535034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.535279 (+ 245us) server_negotiation.cc:407] Connection header received
0808 02:07:41.535416 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.535418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.535517 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.535593 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.535886 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.536348 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.537618 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.537856 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.537942 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.538271 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.538339 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.538396 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.538557 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.538609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:07:41.545475 8745 raft_consensus.cc:1261] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.545728 8570 raft_consensus.cc:1261] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:41.545938 8837 consensus_queue.cc:1035] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.546332 8894 consensus_queue.cc:1035] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:41.566253 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.562571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46591 (local address 127.5.209.66:37183)
0808 02:07:41.562708 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:41.562725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:41.562834 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:41.563164 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:41.563167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:41.563175 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:07:41.563278 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.563284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.564114 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.564120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.564966 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:41.564980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.565576 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.565579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:41.565644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.565672 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.565986 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.565994 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:07:41.566071 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:07:41.566145 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 02:07:41.566289 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:41.562790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:37183 (local address 127.5.209.65:46591)
0808 02:07:41.562931 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:41.562935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:41.562947 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:07:41.563025 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:41.563027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:41.563065 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:41.563146 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:41.563389 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.563999 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.565098 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:41.565475 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:41.565581 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:41.565783 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:41.565867 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:41.565941 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:41.566100 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:07:41.566160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 02:07:41.567026 8881 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:07:43.131706 8178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:43.131860 8178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:43.134174 8178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Sending DeleteTablet for 3 replicas of tablet f11a225235b14f12b7804f742ebd055c
I20260808 02:07:43.134672 8340 tablet_service.cc:1514] Processing DeleteTablet for tablet f11a225235b14f12b7804f742ebd055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260808 02:07:43.134805 8717 tablet_service.cc:1514] Processing DeleteTablet for tablet f11a225235b14f12b7804f742ebd055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34780
I20260808 02:07:43.134845 8541 tablet_service.cc:1514] Processing DeleteTablet for tablet f11a225235b14f12b7804f742ebd055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:57418
I20260808 02:07:43.135159 8936 tablet_replica.cc:330] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: stopping tablet replica
I20260808 02:07:43.135320 8936 raft_consensus.cc:2229] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:43.135320 8938 tablet_replica.cc:330] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: stopping tablet replica
I20260808 02:07:43.135560 8938 raft_consensus.cc:2229] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:43.135653 8936 raft_consensus.cc:2258] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.135691 8938 raft_consensus.cc:2258] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.135913 8937 tablet_replica.cc:330] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: stopping tablet replica
I20260808 02:07:43.136071 8937 raft_consensus.cc:2229] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:43.136186 8937 raft_consensus.cc:2258] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.136468 8938 ts_tablet_manager.cc:1905] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.136734 8936 ts_tablet_manager.cc:1905] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.138525 8937 ts_tablet_manager.cc:1905] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.139616 8936 ts_tablet_manager.cc:1918] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:07:43.139767 8936 log.cc:1196] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/f11a225235b14f12b7804f742ebd055c
I20260808 02:07:43.140219 8936 ts_tablet_manager.cc:1939] T f11a225235b14f12b7804f742ebd055c P 951a0b03757b4f27876054fa1457f8f7: Deleting consensus metadata
I20260808 02:07:43.140867 8938 ts_tablet_manager.cc:1918] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:07:43.140969 8938 log.cc:1196] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/f11a225235b14f12b7804f742ebd055c
I20260808 02:07:43.140980 8162 catalog_manager.cc:4841] TS 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591): tablet f11a225235b14f12b7804f742ebd055c (table test-table [id=3dd8aa5764e6474887973d1a67a18ea6]) successfully deleted
I20260808 02:07:43.141279 8938 ts_tablet_manager.cc:1939] T f11a225235b14f12b7804f742ebd055c P 9bd8ee07e76443e188e1bd23b50f7016: Deleting consensus metadata
I20260808 02:07:43.141662 8937 ts_tablet_manager.cc:1918] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:07:43.141769 8937 log.cc:1196] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/f11a225235b14f12b7804f742ebd055c
I20260808 02:07:43.142040 8937 ts_tablet_manager.cc:1939] T f11a225235b14f12b7804f742ebd055c P 502ee16253df4284ac28e9e4ff8df4f8: Deleting consensus metadata
I20260808 02:07:43.142081 8162 catalog_manager.cc:4841] TS 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751): tablet f11a225235b14f12b7804f742ebd055c (table test-table [id=3dd8aa5764e6474887973d1a67a18ea6]) successfully deleted
I20260808 02:07:43.142706 8163 catalog_manager.cc:4841] TS 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067): tablet f11a225235b14f12b7804f742ebd055c (table test-table [id=3dd8aa5764e6474887973d1a67a18ea6]) successfully deleted
Aug 08 02:07:43 dist-test-slave-bw88 krb5kdc[8121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:07:43 dist-test-slave-bw88 krb5kdc[8121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:43.164287 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:43.155494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.5.209.126:45071)
0808 02:07:43.155785 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:43.155800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:43.155866 (+ 66us) server_negotiation.cc:407] Connection header received
0808 02:07:43.155990 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:43.155992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:43.156035 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:43.156111 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:43.156771 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:43.157381 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:43.158219 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:43.158458 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:43.158566 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:43.160950 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:43.160953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:43.160955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:43.160977 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:43.162551 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:43.162999 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:43.163003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:43.163005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:43.163097 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:43.163373 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:43.163376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:43.163378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:43.163583 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:43.163725 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:43.163976 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 02:07:43.164111 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260808 02:07:43.166893 8178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:43.167301 8178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:43.172093 8340 tablet_service.cc:1467] Processing CreateTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 (DEFAULT_TABLE table=test-table [id=ead8a61831eb4515a2446314371c21ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.172108 8541 tablet_service.cc:1467] Processing CreateTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 (DEFAULT_TABLE table=test-table [id=ead8a61831eb4515a2446314371c21ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.172215 8717 tablet_service.cc:1467] Processing CreateTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 (DEFAULT_TABLE table=test-table [id=ead8a61831eb4515a2446314371c21ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.172438 8541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4c88d5ea4541f69aea7e43c5692ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.172446 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4c88d5ea4541f69aea7e43c5692ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.172475 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4c88d5ea4541f69aea7e43c5692ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.175156 8949 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Bootstrap starting.
I20260808 02:07:43.175563 8948 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Bootstrap starting.
I20260808 02:07:43.176079 8949 tablet_bootstrap.cc:654] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.176285 8948 tablet_bootstrap.cc:654] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.176713 8950 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Bootstrap starting.
I20260808 02:07:43.177511 8948 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: No bootstrap required, opened a new log
I20260808 02:07:43.177582 8950 tablet_bootstrap.cc:654] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.177618 8948 ts_tablet_manager.cc:1397] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:43.178040 8949 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: No bootstrap required, opened a new log
I20260808 02:07:43.178107 8948 raft_consensus.cc:348] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.178233 8948 raft_consensus.cc:374] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.178272 8948 raft_consensus.cc:729] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.178200 8949 ts_tablet_manager.cc:1397] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:43.178416 8948 consensus_queue.cc:260] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.178679 8950 tablet_bootstrap.cc:492] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: No bootstrap required, opened a new log
I20260808 02:07:43.178707 8948 ts_tablet_manager.cc:1428] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:43.178766 8950 ts_tablet_manager.cc:1397] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:43.179056 8949 raft_consensus.cc:348] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.179172 8950 raft_consensus.cc:348] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.179286 8950 raft_consensus.cc:374] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.179317 8950 raft_consensus.cc:729] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.179232 8949 raft_consensus.cc:374] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.179437 8950 consensus_queue.cc:260] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.179698 8950 ts_tablet_manager.cc:1428] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:43.179973 8949 raft_consensus.cc:729] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ee16253df4284ac28e9e4ff8df4f8, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.180227 8949 consensus_queue.cc:260] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.180728 8949 ts_tablet_manager.cc:1428] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:43.475445 8926 raft_consensus.cc:482] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:43.475663 8926 raft_consensus.cc:504] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.476086 8926 leader_election.cc:288] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:43.476668 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4c88d5ea4541f69aea7e43c5692ee2" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8" is_pre_election: true
I20260808 02:07:43.476670 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4c88d5ea4541f69aea7e43c5692ee2" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016" is_pre_election: true
I20260808 02:07:43.476854 8744 raft_consensus.cc:2454] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:43.476854 8570 raft_consensus.cc:2454] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:43.477231 8303 leader_election.cc:302] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 951a0b03757b4f27876054fa1457f8f7; no voters:
I20260808 02:07:43.477485 8926 raft_consensus.cc:2781] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:43.477577 8926 raft_consensus.cc:482] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:43.477632 8926 raft_consensus.cc:3037] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.478870 8926 raft_consensus.cc:504] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.479246 8926 leader_election.cc:288] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Requested vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:43.479657 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4c88d5ea4541f69aea7e43c5692ee2" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8"
I20260808 02:07:43.479657 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4c88d5ea4541f69aea7e43c5692ee2" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016"
I20260808 02:07:43.479797 8744 raft_consensus.cc:3037] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.479797 8570 raft_consensus.cc:3037] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.481026 8744 raft_consensus.cc:2454] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:43.481025 8570 raft_consensus.cc:2454] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:43.481395 8303 leader_election.cc:302] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 951a0b03757b4f27876054fa1457f8f7; no voters:
I20260808 02:07:43.481652 8926 raft_consensus.cc:2781] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:43.481766 8926 raft_consensus.cc:686] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Running, Role: LEADER
I20260808 02:07:43.481938 8926 consensus_queue.cc:237] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.483762 8179 catalog_manager.cc:5495] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "951a0b03757b4f27876054fa1457f8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:43.548898 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:43.545783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.5.209.65:46591)
0808 02:07:43.546048 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:43.546055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:43.546068 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:07:43.546108 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:43.546111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:43.546153 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:43.546232 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:43.546479 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:43.547024 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:43.547850 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:43.548090 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:43.548172 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:43.548425 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:43.548506 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:43.548573 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:43.548702 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:07:43.548766 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260808 02:07:43.555693 8744 raft_consensus.cc:1261] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:43.555787 8570 raft_consensus.cc:1261] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:43.556150 8903 consensus_queue.cc:1035] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:43.556348 8926 consensus_queue.cc:1035] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:43.584139 8179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:43.584350 8179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:43.586411 8179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Sending DeleteTablet for 3 replicas of tablet ae4c88d5ea4541f69aea7e43c5692ee2
I20260808 02:07:43.587040 8340 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260808 02:07:43.587236 8936 tablet_replica.cc:330] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: stopping tablet replica
I20260808 02:07:43.587363 8936 raft_consensus.cc:2229] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:43.587327 8717 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34780
I20260808 02:07:43.587554 8937 tablet_replica.cc:330] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: stopping tablet replica
I20260808 02:07:43.587561 8936 raft_consensus.cc:2258] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.587694 8937 raft_consensus.cc:2229] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:43.587822 8937 raft_consensus.cc:2258] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.588037 8541 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4c88d5ea4541f69aea7e43c5692ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:43 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:57418
I20260808 02:07:43.588249 8938 tablet_replica.cc:330] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: stopping tablet replica
I20260808 02:07:43.588380 8938 raft_consensus.cc:2229] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:43.588471 8936 ts_tablet_manager.cc:1905] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.588505 8938 raft_consensus.cc:2258] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:43.589136 8937 ts_tablet_manager.cc:1905] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.589079 8179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:43.589546 8179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:43.589639 8938 ts_tablet_manager.cc:1905] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:43.591598 8936 ts_tablet_manager.cc:1918] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:43.591674 8936 log.cc:1196] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/ae4c88d5ea4541f69aea7e43c5692ee2
I20260808 02:07:43.592059 8936 ts_tablet_manager.cc:1939] T ae4c88d5ea4541f69aea7e43c5692ee2 P 951a0b03757b4f27876054fa1457f8f7: Deleting consensus metadata
I20260808 02:07:43.592123 8937 ts_tablet_manager.cc:1918] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:43.592201 8937 log.cc:1196] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/ae4c88d5ea4541f69aea7e43c5692ee2
I20260808 02:07:43.592315 8938 ts_tablet_manager.cc:1918] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:07:43.592389 8938 log.cc:1196] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/ae4c88d5ea4541f69aea7e43c5692ee2
I20260808 02:07:43.592478 8937 ts_tablet_manager.cc:1939] T ae4c88d5ea4541f69aea7e43c5692ee2 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting consensus metadata
I20260808 02:07:43.592675 8938 ts_tablet_manager.cc:1939] T ae4c88d5ea4541f69aea7e43c5692ee2 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting consensus metadata
I20260808 02:07:43.592787 8162 catalog_manager.cc:4841] TS 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591): tablet ae4c88d5ea4541f69aea7e43c5692ee2 (table test-table [id=ead8a61831eb4515a2446314371c21ae]) successfully deleted
I20260808 02:07:43.593393 8162 catalog_manager.cc:4841] TS 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751): tablet ae4c88d5ea4541f69aea7e43c5692ee2 (table test-table [id=ead8a61831eb4515a2446314371c21ae]) successfully deleted
I20260808 02:07:43.593569 8163 catalog_manager.cc:4841] TS 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067): tablet ae4c88d5ea4541f69aea7e43c5692ee2 (table test-table [id=ead8a61831eb4515a2446314371c21ae]) successfully deleted
I20260808 02:07:43.594499 8541 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6ea83a3a734f47847c3896750bd539 (DEFAULT_TABLE table=test-table [id=575061ea4fb74997a31de576665e2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.594470 8340 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6ea83a3a734f47847c3896750bd539 (DEFAULT_TABLE table=test-table [id=575061ea4fb74997a31de576665e2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.594785 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6ea83a3a734f47847c3896750bd539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.594780 8541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6ea83a3a734f47847c3896750bd539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.597100 8948 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: Bootstrap starting.
I20260808 02:07:43.597584 8717 tablet_service.cc:1467] Processing CreateTablet for tablet 9e6ea83a3a734f47847c3896750bd539 (DEFAULT_TABLE table=test-table [id=575061ea4fb74997a31de576665e2ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:43.597851 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e6ea83a3a734f47847c3896750bd539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:43.598075 8948 tablet_bootstrap.cc:654] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.598739 8950 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Bootstrap starting.
I20260808 02:07:43.599507 8948 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: No bootstrap required, opened a new log
I20260808 02:07:43.599592 8950 tablet_bootstrap.cc:654] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.599611 8948 ts_tablet_manager.cc:1397] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:43.600056 8949 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Bootstrap starting.
I20260808 02:07:43.600260 8948 raft_consensus.cc:348] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.600373 8948 raft_consensus.cc:374] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.600396 8948 raft_consensus.cc:729] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.600544 8948 consensus_queue.cc:260] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.600833 8948 ts_tablet_manager.cc:1428] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:07:43.601028 8949 tablet_bootstrap.cc:654] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:43.602368 8950 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: No bootstrap required, opened a new log
I20260808 02:07:43.602478 8950 ts_tablet_manager.cc:1397] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 02:07:43.602888 8950 raft_consensus.cc:348] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.603011 8950 raft_consensus.cc:374] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.603053 8950 raft_consensus.cc:729] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8ee07e76443e188e1bd23b50f7016, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.603120 8949 tablet_bootstrap.cc:492] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: No bootstrap required, opened a new log
I20260808 02:07:43.603214 8949 ts_tablet_manager.cc:1397] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:43.603189 8950 consensus_queue.cc:260] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.603435 8950 ts_tablet_manager.cc:1428] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:07:43.603806 8949 raft_consensus.cc:348] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.603938 8949 raft_consensus.cc:374] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:43.603993 8949 raft_consensus.cc:729] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ee16253df4284ac28e9e4ff8df4f8, State: Initialized, Role: FOLLOWER
I20260808 02:07:43.604138 8949 consensus_queue.cc:260] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.604444 8949 ts_tablet_manager.cc:1428] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:07:43.783741 8926 raft_consensus.cc:482] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:43.783978 8926 raft_consensus.cc:504] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.784502 8926 leader_election.cc:288] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:43.785059 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6ea83a3a734f47847c3896750bd539" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016" is_pre_election: true
I20260808 02:07:43.785084 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6ea83a3a734f47847c3896750bd539" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8" is_pre_election: true
I20260808 02:07:43.785246 8744 raft_consensus.cc:2454] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:43.785246 8570 raft_consensus.cc:2454] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 0.
I20260808 02:07:43.785614 8302 leader_election.cc:302] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951a0b03757b4f27876054fa1457f8f7, 9bd8ee07e76443e188e1bd23b50f7016; no voters:
I20260808 02:07:43.785781 8926 raft_consensus.cc:2781] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:43.785848 8926 raft_consensus.cc:482] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:43.785904 8926 raft_consensus.cc:3037] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.786947 8926 raft_consensus.cc:504] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.787330 8926 leader_election.cc:288] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Requested vote from peers 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751), 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067)
I20260808 02:07:43.787660 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6ea83a3a734f47847c3896750bd539" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd8ee07e76443e188e1bd23b50f7016"
I20260808 02:07:43.787734 8744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e6ea83a3a734f47847c3896750bd539" candidate_uuid: "951a0b03757b4f27876054fa1457f8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ee16253df4284ac28e9e4ff8df4f8"
I20260808 02:07:43.787804 8570 raft_consensus.cc:3037] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.787868 8744 raft_consensus.cc:3037] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:43.788794 8744 raft_consensus.cc:2454] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:43.789081 8303 leader_election.cc:302] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ee16253df4284ac28e9e4ff8df4f8, 951a0b03757b4f27876054fa1457f8f7; no voters:
I20260808 02:07:43.789263 8926 raft_consensus.cc:2781] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:43.789404 8926 raft_consensus.cc:686] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 951a0b03757b4f27876054fa1457f8f7, State: Running, Role: LEADER
I20260808 02:07:43.789505 8570 raft_consensus.cc:2454] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 951a0b03757b4f27876054fa1457f8f7 in term 1.
I20260808 02:07:43.789561 8926 consensus_queue.cc:237] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } }
I20260808 02:07:43.791507 8179 catalog_manager.cc:5495] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "951a0b03757b4f27876054fa1457f8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951a0b03757b4f27876054fa1457f8f7" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:43.845757 8745 raft_consensus.cc:1261] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:43.846113 8570 raft_consensus.cc:1261] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Refusing update from remote peer 951a0b03757b4f27876054fa1457f8f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:43.846213 8926 consensus_queue.cc:1035] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ee16253df4284ac28e9e4ff8df4f8" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:43.846637 8926 consensus_queue.cc:1035] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd8ee07e76443e188e1bd23b50f7016" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:07:43.860644 8911 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:07:45.410874 8178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:45.411059 8178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:45.413429 8178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe6aaf3f5ddf407ebf5566def14e61aa: Sending DeleteTablet for 3 replicas of tablet 9e6ea83a3a734f47847c3896750bd539
I20260808 02:07:45.413947 8340 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e6ea83a3a734f47847c3896750bd539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260808 02:07:45.413981 8541 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e6ea83a3a734f47847c3896750bd539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:57418
I20260808 02:07:45.414177 8717 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e6ea83a3a734f47847c3896750bd539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34780
I20260808 02:07:45.414317 8999 tablet_replica.cc:330] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: stopping tablet replica
I20260808 02:07:45.414462 8999 raft_consensus.cc:2229] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:45.414564 9000 tablet_replica.cc:330] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: stopping tablet replica
I20260808 02:07:45.414707 8998 tablet_replica.cc:330] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7: stopping tablet replica
I20260808 02:07:45.414738 9000 raft_consensus.cc:2229] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:45.414855 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 8279
I20260808 02:07:45.414866 9000 raft_consensus.cc:2258] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:45.414892 8998 raft_consensus.cc:2229] T 9e6ea83a3a734f47847c3896750bd539 P 951a0b03757b4f27876054fa1457f8f7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:45.420195 8999 raft_consensus.cc:2258] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:45.436299 8999 ts_tablet_manager.cc:1905] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:45.436343 9000 ts_tablet_manager.cc:1905] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:45.438858 9000 ts_tablet_manager.cc:1918] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:07:45.438961 9000 log.cc:1196] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/9e6ea83a3a734f47847c3896750bd539
I20260808 02:07:45.439273 9000 ts_tablet_manager.cc:1939] T 9e6ea83a3a734f47847c3896750bd539 P 502ee16253df4284ac28e9e4ff8df4f8: Deleting consensus metadata
I20260808 02:07:45.439545 8999 ts_tablet_manager.cc:1918] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:07:45.439640 8999 log.cc:1196] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/9e6ea83a3a734f47847c3896750bd539
I20260808 02:07:45.439971 8999 ts_tablet_manager.cc:1939] T 9e6ea83a3a734f47847c3896750bd539 P 9bd8ee07e76443e188e1bd23b50f7016: Deleting consensus metadata
I20260808 02:07:45.440181 8163 catalog_manager.cc:4841] TS 502ee16253df4284ac28e9e4ff8df4f8 (127.5.209.67:41067): tablet 9e6ea83a3a734f47847c3896750bd539 (table test-table [id=575061ea4fb74997a31de576665e2ea4]) successfully deleted
I20260808 02:07:45.440877 8162 catalog_manager.cc:4841] TS 9bd8ee07e76443e188e1bd23b50f7016 (127.5.209.66:45751): tablet 9e6ea83a3a734f47847c3896750bd539 (table test-table [id=575061ea4fb74997a31de576665e2ea4]) successfully deleted
W20260808 02:07:45.444850 8162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:46591 (error 108)
I20260808 02:07:45.445216 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 8461
I20260808 02:07:45.451015 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:45.449780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:46591 (local address 127.0.0.1:48204)
0808 02:07:45.450786 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:45.450879 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":474,"threads_started":1}
W20260808 02:07:45.451160 8162 catalog_manager.cc:4584] TS 951a0b03757b4f27876054fa1457f8f7 (127.5.209.65:46591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e6ea83a3a734f47847c3896750bd539: Network error: Client connection negotiation failed: client connection to 127.5.209.65:46591: connect: Connection refused (error 111)
I20260808 02:07:45.459532 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 8641
I20260808 02:07:45.471005 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 8135
2026-08-08T02:07:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6783 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:45 dist-test-slave-bw88 krb5kdc[9006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:45 dist-test-slave-bw88 krb5kdc[9006](info): set up 2 sockets
Aug 08 02:07:45 dist-test-slave-bw88 krb5kdc[9006](info): commencing operation
krb5kdc: starting...
W20260808 02:07:46.666936 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:46 dist-test-slave-bw88 krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:46.792268 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35973
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:46.905121 9020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:46.905442 9020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:46.905534 9020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:46.909516 9020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:46.909605 9020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:46.909643 9020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:46.909699 9020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:46.909731 9020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:46.914944 9020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:46.916028 9020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:46.917050 9020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:46.925774 9033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:46.925801 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:46.926012 9020 server_base.cc:1034] running on GCE node
W20260808 02:07:46.925806 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:46.926741 9020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:46.927893 9020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:46.929056 9020 hybrid_clock.cc:648] HybridClock initialized: now 1786154866929041 us; error 26 us; skew 500 ppm
Aug 08 02:07:46 dist-test-slave-bw88 krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:46.932107 9020 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:46.933039 9020 webserver.cc:460] Webserver started at http://127.5.209.126:33687/ using document root <none> and password file <none>
I20260808 02:07:46.933698 9020 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:46.933765 9020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:46.934020 9020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:46.936149 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "2f9b4f13440f429381d8e6f47e61f937"
format_stamp: "Formatted at 2026-08-08 02:07:46 on dist-test-slave-bw88"
server_key: "c6f56cad42a0ce93863fb69112d3f50a"
server_key_iv: "b6dc4c0ca04774a1af292eb8b0f432cd"
server_key_version: "encryptionkey@0"
I20260808 02:07:46.936739 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "2f9b4f13440f429381d8e6f47e61f937"
format_stamp: "Formatted at 2026-08-08 02:07:46 on dist-test-slave-bw88"
server_key: "c6f56cad42a0ce93863fb69112d3f50a"
server_key_iv: "b6dc4c0ca04774a1af292eb8b0f432cd"
server_key_version: "encryptionkey@0"
I20260808 02:07:46.940541 9020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 02:07:46.943077 9041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:46.944264 9020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:46.944401 9020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "2f9b4f13440f429381d8e6f47e61f937"
format_stamp: "Formatted at 2026-08-08 02:07:46 on dist-test-slave-bw88"
server_key: "c6f56cad42a0ce93863fb69112d3f50a"
server_key_iv: "b6dc4c0ca04774a1af292eb8b0f432cd"
server_key_version: "encryptionkey@0"
I20260808 02:07:46.944507 9020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:46.966909 9020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:46.969871 9020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:46.970088 9020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:46.977882 9020 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:35973
I20260808 02:07:46.977909 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:35973 every 8 connection(s)
I20260808 02:07:46.978863 9020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:46.981876 9107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:46.987804 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937: Bootstrap starting.
I20260808 02:07:46.988317 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9020
I20260808 02:07:46.988444 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:46.988734 5957 external_mini_cluster.cc:1428] Setting key ecdf4687688ae4b9ac159cbb38f9df20
I20260808 02:07:46.990691 9107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:46.991712 9107 log.cc:824] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:46.993762 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937: No bootstrap required, opened a new log
I20260808 02:07:46.997309 9107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } }
I20260808 02:07:46.997596 9107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:46.997675 9107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9b4f13440f429381d8e6f47e61f937, State: Initialized, Role: FOLLOWER
I20260808 02:07:46.998131 9107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } }
I20260808 02:07:46.998304 9107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:46.998386 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:46.998509 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 02:07:46 dist-test-slave-bw88 krb5kdc[9006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:46.999503 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } }
I20260808 02:07:46.999872 9107 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9b4f13440f429381d8e6f47e61f937; no voters:
I20260808 02:07:47.000269 9107 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:47.000378 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:47.000663 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [term 1 LEADER]: Becoming Leader. State: Replica: 2f9b4f13440f429381d8e6f47e61f937, State: Running, Role: LEADER
I20260808 02:07:47.001065 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } }
I20260808 02:07:47.001776 9107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:47.002374 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9b4f13440f429381d8e6f47e61f937. Latest consensus state: current_term: 1 leader_uuid: "2f9b4f13440f429381d8e6f47e61f937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } } }
I20260808 02:07:47.002580 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:47.003003 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9b4f13440f429381d8e6f47e61f937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9b4f13440f429381d8e6f47e61f937" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35973 } } }
I20260808 02:07:47.003129 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:47.003060 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:47.006243 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:46.990167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.5.209.126:35973)
0808 02:07:46.990644 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:46.990668 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:46.990716 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:07:46.991706 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:46.991734 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:46.992190 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:46.992558 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:46.993502 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:46.994666 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:46.995622 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:46.996137 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:46.996329 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:46.999399 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:46.999407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:46.999427 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:46.999462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:47.001652 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.002194 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.002200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.002206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.002344 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.003463 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.003467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.003469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.003979 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:47.004247 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:47.004937 (+ 690us) server_negotiation.cc:299] Negotiation successful
0808 02:07:47.005256 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":113,"threads_started":1}
I20260808 02:07:47.007073 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:47.013532 9122 catalog_manager.cc:1347] Generated new cluster ID: f8c401d0a25e445fbe53373768e51e89
I20260808 02:07:47.013602 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:47.031169 9122 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:47.032323 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:47.039983 9122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9b4f13440f429381d8e6f47e61f937: Generated new TSK 0
I20260808 02:07:47.040683 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:47.074146 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:47.196730 9145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:47.196982 9145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:47.197024 9145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:47.201279 9145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:47.201397 9145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:47.201506 9145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:47.206872 9145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:47.207923 9145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:47.208976 9145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:47.218737 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:47.218802 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:47.218843 9145 server_base.cc:1034] running on GCE node
W20260808 02:07:47.218802 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:47.219341 9145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:47.219991 9145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:47.221161 9145 hybrid_clock.cc:648] HybridClock initialized: now 1786154867221120 us; error 45 us; skew 500 ppm
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:47.224037 9145 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:47.224809 9145 webserver.cc:460] Webserver started at http://127.5.209.65:34849/ using document root <none> and password file <none>
I20260808 02:07:47.225317 9145 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:47.225435 9145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:47.225606 9145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:47.227546 9145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "1d76659b78fa42e4b359208bc9a7e104"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "9a7aff85443ea6a4b4bb2714b0631183"
server_key_iv: "e17f99006593b9ffe945ccc621635307"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.228089 9145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "1d76659b78fa42e4b359208bc9a7e104"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "9a7aff85443ea6a4b4bb2714b0631183"
server_key_iv: "e17f99006593b9ffe945ccc621635307"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.231843 9145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:07:47.234341 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.235584 9145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:47.235705 9145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "1d76659b78fa42e4b359208bc9a7e104"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "9a7aff85443ea6a4b4bb2714b0631183"
server_key_iv: "e17f99006593b9ffe945ccc621635307"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.235805 9145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:47.252732 9145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:47.255999 9145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:47.256203 9145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:47.256831 9145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:47.257707 9145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:47.257755 9145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.257823 9145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:47.257855 9145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.267186 9145 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:45775
I20260808 02:07:47.267263 9301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:45775 every 8 connection(s)
I20260808 02:07:47.268118 9145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:47.272526 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9145
I20260808 02:07:47.272655 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:47.272927 5957 external_mini_cluster.cc:1428] Setting key b050d5af6e148c8e9e910d3e9a493ba9
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:47.283435 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.270032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:38447 (local address 127.5.209.126:35973)
0808 02:07:47.270201 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:47.270206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:47.271478 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 02:07:47.272361 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:47.272368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:47.272437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:47.272625 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:47.274017 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.274714 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.276457 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.276695 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.276779 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.279313 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:47.279319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:47.279325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:47.279359 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:47.281178 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.281734 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.281741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.281744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.281849 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.282206 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.282211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.282213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.282459 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:47.282565 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:47.283059 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 02:07:47.283220 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:07:47.284049 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.270433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35973 (local address 127.5.209.65:38447)
0808 02:07:47.271320 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:47.271362 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:47.272126 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:47.272827 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:47.272834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:47.273237 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:47.273855 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.273870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.274855 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.274862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.276332 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.276349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.276829 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.276832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.276973 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.277594 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:47.277621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:47.279142 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:47.281297 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.281302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.281313 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.281630 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.281948 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.281950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.281952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.282096 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.282574 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:47.282578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:47.282807 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 02:07:47.282952 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 02:07:47.283204 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":555,"threads_started":1}
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:47.285696 9302 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35973
I20260808 02:07:47.285948 9302 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:47.286430 9302 heartbeater.cc:511] Master 127.5.209.126:35973 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:07:47.288058 9062 ts_manager.cc:194] Registered new tserver with Master: 1d76659b78fa42e4b359208bc9a7e104 (127.5.209.65:45775)
I20260808 02:07:47.289762 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:38447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:47.304245 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:47.422574 9307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:47.422803 9307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:47.422845 9307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:47.426817 9307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:47.426868 9307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:47.426954 9307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:47.432216 9307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:47.433236 9307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:47.434265 9307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:47.444062 9313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:47.444181 9312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:47.444368 9307 server_base.cc:1034] running on GCE node
W20260808 02:07:47.444579 9315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:47.445004 9307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:47.445668 9307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:47.446858 9307 hybrid_clock.cc:648] HybridClock initialized: now 1786154867446830 us; error 51 us; skew 500 ppm
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:47.450201 9307 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:47.451419 9307 webserver.cc:460] Webserver started at http://127.5.209.66:39021/ using document root <none> and password file <none>
I20260808 02:07:47.452198 9307 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:47.452301 9307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:47.452574 9307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:47.455763 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "36c5654fae8445f8b81b14a82a27f363"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "027378b8af87bd53783c68bc4371bf0c"
server_key_iv: "e50ad0135b298a5d9187ed9f8adbbe8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.456605 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "36c5654fae8445f8b81b14a82a27f363"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "027378b8af87bd53783c68bc4371bf0c"
server_key_iv: "e50ad0135b298a5d9187ed9f8adbbe8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.461966 9307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:07:47.465524 9324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.467137 9307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:07:47.467265 9307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "36c5654fae8445f8b81b14a82a27f363"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "027378b8af87bd53783c68bc4371bf0c"
server_key_iv: "e50ad0135b298a5d9187ed9f8adbbe8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.467396 9307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:47.504528 9307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:47.508044 9307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:47.508265 9307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:47.508875 9307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:47.509814 9307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:47.509887 9307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.509960 9307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:47.510001 9307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.520604 9307 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:34187
I20260808 02:07:47.520637 9458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:34187 every 8 connection(s)
I20260808 02:07:47.521618 9307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:47.531095 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9307
I20260808 02:07:47.531217 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:47.531873 5957 external_mini_cluster.cc:1428] Setting key 2859529285ad977952164296695b9526
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:47.537797 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.524019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:60177 (local address 127.5.209.126:35973)
0808 02:07:47.524181 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:47.524187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:47.525083 (+ 896us) server_negotiation.cc:407] Connection header received
0808 02:07:47.526535 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:47.526540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:47.526612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:47.526729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:47.527937 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.528925 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.530163 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.530638 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.530771 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.533670 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:47.533673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:47.533675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:47.533699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:47.535187 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.535736 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.535740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.535741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.535822 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.536244 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.536250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.536254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.536511 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:47.536608 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:47.537390 (+ 782us) server_negotiation.cc:299] Negotiation successful
0808 02:07:47.537548 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:07:47.538585 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.524436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35973 (local address 127.5.209.66:60177)
0808 02:07:47.524912 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:47.524966 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:47.526301 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:47.526880 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:47.526886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:47.527244 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:47.527787 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.527802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.529064 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.529067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.530025 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.530036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.530789 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.530792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.531011 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.531664 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:47.531686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:47.533488 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:47.535326 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.535332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.535347 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.535624 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.535918 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.535921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.535923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.536100 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.536688 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:47.536697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:47.537034 (+ 337us) client_negotiation.cc:769] Sending connection context
0808 02:07:47.537246 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 02:07:47.537638 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260808 02:07:47.539912 9459 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35973
I20260808 02:07:47.540174 9459 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:47.540721 9459 heartbeater.cc:511] Master 127.5.209.126:35973 requested a full tablet report, sending...
I20260808 02:07:47.541867 9062 ts_manager.cc:194] Registered new tserver with Master: 36c5654fae8445f8b81b14a82a27f363 (127.5.209.66:34187)
I20260808 02:07:47.542440 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:60177
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:47.567971 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:47.696899 9464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:47.697170 9464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:47.697237 9464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:47.701140 9464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:47.701206 9464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:47.701283 9464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:47.706597 9464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:47.707546 9464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:47.708551 9464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:47.718344 9470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:47.718339 9475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:47.718375 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:47.718711 9464 server_base.cc:1034] running on GCE node
I20260808 02:07:47.719203 9464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:47.719812 9464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:47.721009 9464 hybrid_clock.cc:648] HybridClock initialized: now 1786154867721009 us; error 41 us; skew 500 ppm
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:47.723950 9464 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:47.724825 9464 webserver.cc:460] Webserver started at http://127.5.209.67:33651/ using document root <none> and password file <none>
I20260808 02:07:47.725425 9464 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:47.725556 9464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:47.725797 9464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:47.727749 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "64b9d565fb464e13962bfb2a24da4c02"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "378c9675dc8d0e83d8206cc70bfd170d"
server_key_iv: "a1600045f97ded1f97a01b6711347cbc"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.728312 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "64b9d565fb464e13962bfb2a24da4c02"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "378c9675dc8d0e83d8206cc70bfd170d"
server_key_iv: "a1600045f97ded1f97a01b6711347cbc"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.732080 9464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 02:07:47.734586 9483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.735754 9464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:47.735886 9464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "64b9d565fb464e13962bfb2a24da4c02"
format_stamp: "Formatted at 2026-08-08 02:07:47 on dist-test-slave-bw88"
server_key: "378c9675dc8d0e83d8206cc70bfd170d"
server_key_iv: "a1600045f97ded1f97a01b6711347cbc"
server_key_version: "encryptionkey@0"
I20260808 02:07:47.735991 9464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:47.754171 9464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:47.756963 9464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:47.757161 9464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:47.757802 9464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:47.758633 9464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:47.758702 9464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.758776 9464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:47.758821 9464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:47.768088 9464 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:38879
I20260808 02:07:47.768122 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:38879 every 8 connection(s)
I20260808 02:07:47.769068 9464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:47.775635 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9464
I20260808 02:07:47.775764 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:47.776098 5957 external_mini_cluster.cc:1428] Setting key 1da6bc5ff6a724a9f20a46ed21d73d27
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:47.783003 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.770935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:41429 (local address 127.5.209.126:35973)
0808 02:07:47.771098 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:47.771112 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:47.771901 (+ 789us) server_negotiation.cc:407] Connection header received
0808 02:07:47.773457 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:47.773460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:47.773509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:47.773581 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:47.774706 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.775236 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.776787 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.777044 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.777143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.779730 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:47.779733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:47.779735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:47.779758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:47.781161 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.781721 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.781724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.781725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.781801 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:47.782124 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:47.782126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:47.782127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:47.782288 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:47.782362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:47.782793 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 02:07:47.782886 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:07:47.783812 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:47.771301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35973 (local address 127.5.209.67:41429)
0808 02:07:47.771754 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:47.771796 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:47.773228 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:47.773712 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:47.773718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:47.774051 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:47.774571 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.774585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.775337 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.775340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.776650 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:47.776661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:47.777197 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:47.777200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:47.777453 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:47.778022 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:47.778038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:47.779585 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:47.781274 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.781279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.781290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.781626 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.781878 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:47.781880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:47.781881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:47.782035 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:47.782373 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:47.782378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:47.782591 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 02:07:47.782734 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 02:07:47.782984 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260808 02:07:47.784857 9622 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35973
I20260808 02:07:47.785081 9622 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:47.785589 9622 heartbeater.cc:511] Master 127.5.209.126:35973 requested a full tablet report, sending...
I20260808 02:07:47.786626 9062 ts_manager.cc:194] Registered new tserver with Master: 64b9d565fb464e13962bfb2a24da4c02 (127.5.209.67:38879)
I20260808 02:07:47.787163 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:41429
I20260808 02:07:47.790026 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:07:47.805977 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:07:47.804454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.5.209.126:35973)
0808 02:07:47.804596 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:47.804600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:47.804799 (+ 199us) server_negotiation.cc:407] Connection header received
0808 02:07:47.804915 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:47.804918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:47.804963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:47.805045 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:47.805800 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43398: BlockingRecv error: recv got EOF from 127.0.0.1:43398 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:07:47.807235 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9145
I20260808 02:07:47.835417 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9307
I20260808 02:07:47.845443 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9464
I20260808 02:07:47.855049 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9020
2026-08-08T02:07:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2383 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9640](info): set up 2 sockets
Aug 08 02:07:47 dist-test-slave-bw88 krb5kdc[9640](info): commencing operation
krb5kdc: starting...
W20260808 02:07:49.052798 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:49.192175 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35665
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:49.307497 9654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:49.307814 9654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:49.307895 9654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:49.311913 9654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:49.312002 9654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:49.312042 9654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:49.312098 9654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:49.312137 9654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:49.317453 9654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:49.318548 9654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:49.319615 9654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:49.328397 9659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:49.328372 9663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:49.328389 9661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:49.328653 9654 server_base.cc:1034] running on GCE node
I20260808 02:07:49.329283 9654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:49.330404 9654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:49.331607 9654 hybrid_clock.cc:648] HybridClock initialized: now 1786154869331592 us; error 25 us; skew 500 ppm
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154869, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:49.334560 9654 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:49.335507 9654 webserver.cc:460] Webserver started at http://127.5.209.126:40309/ using document root <none> and password file <none>
I20260808 02:07:49.336082 9654 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:49.336174 9654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:49.336405 9654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:49.338542 9654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "bc908ce265d641ab9b94c73d73ab5e84"
format_stamp: "Formatted at 2026-08-08 02:07:49 on dist-test-slave-bw88"
server_key: "7794ead13c9e16b02f0420a4ad10f57f"
server_key_iv: "84f2a7f3e20109d356f72fc77dac4469"
server_key_version: "encryptionkey@0"
I20260808 02:07:49.339135 9654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "bc908ce265d641ab9b94c73d73ab5e84"
format_stamp: "Formatted at 2026-08-08 02:07:49 on dist-test-slave-bw88"
server_key: "7794ead13c9e16b02f0420a4ad10f57f"
server_key_iv: "84f2a7f3e20109d356f72fc77dac4469"
server_key_version: "encryptionkey@0"
I20260808 02:07:49.342927 9654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:49.345320 9670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:49.346668 9654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:49.346817 9654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "bc908ce265d641ab9b94c73d73ab5e84"
format_stamp: "Formatted at 2026-08-08 02:07:49 on dist-test-slave-bw88"
server_key: "7794ead13c9e16b02f0420a4ad10f57f"
server_key_iv: "84f2a7f3e20109d356f72fc77dac4469"
server_key_version: "encryptionkey@0"
I20260808 02:07:49.346923 9654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:49.360728 9654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:49.363911 9654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:49.364126 9654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:49.371822 9654 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:35665
I20260808 02:07:49.372143 9732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:35665 every 8 connection(s)
I20260808 02:07:49.372853 9654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:49.375818 9733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:49.378902 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9654
I20260808 02:07:49.379057 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:49.379346 5957 external_mini_cluster.cc:1428] Setting key 5dbec0fb16b43c9a052e0a8e873adf55
I20260808 02:07:49.382047 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84: Bootstrap starting.
I20260808 02:07:49.384716 9733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:49.385666 9733 log.cc:824] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:49.387959 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84: No bootstrap required, opened a new log
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:49.391608 9733 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } }
I20260808 02:07:49.391880 9733 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:49.391969 9733 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc908ce265d641ab9b94c73d73ab5e84, State: Initialized, Role: FOLLOWER
I20260808 02:07:49.392539 9733 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } }
I20260808 02:07:49.392728 9733 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:49.392822 9733 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:49.392935 9733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:49.394002 9733 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } }
I20260808 02:07:49.394420 9733 leader_election.cc:302] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc908ce265d641ab9b94c73d73ab5e84; no voters:
I20260808 02:07:49.394845 9733 leader_election.cc:288] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:49.394961 9738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:49.395267 9738 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [term 1 LEADER]: Becoming Leader. State: Replica: bc908ce265d641ab9b94c73d73ab5e84, State: Running, Role: LEADER
I20260808 02:07:49.395679 9738 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } }
I20260808 02:07:49.396436 9733 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:49.396826 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:49.380896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.5.209.126:35665)
0808 02:07:49.381330 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:49.381392 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:49.381438 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:07:49.382092 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:49.382133 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:49.382498 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:49.382782 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:49.383817 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:49.385050 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:49.386288 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:49.386803 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:49.386990 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:49.389676 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:49.389681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:49.389697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:49.389724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:49.393268 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:49.393753 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:49.393759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:49.393766 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:49.393889 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:49.394211 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:49.394214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:49.394216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:49.394822 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:49.395070 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:49.395407 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 02:07:49.395753 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260808 02:07:49.396908 9738 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc908ce265d641ab9b94c73d73ab5e84. Latest consensus state: current_term: 1 leader_uuid: "bc908ce265d641ab9b94c73d73ab5e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } } }
I20260808 02:07:49.397169 9738 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:49.397414 9740 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc908ce265d641ab9b94c73d73ab5e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc908ce265d641ab9b94c73d73ab5e84" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35665 } } }
I20260808 02:07:49.397553 9740 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:49.398172 9745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:49.400772 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:07:49.401922 9745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:49.407721 9745 catalog_manager.cc:1347] Generated new cluster ID: 5c497bdef156423587507f94ce5aa5c0
I20260808 02:07:49.407788 9745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:49.428639 9745 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:49.430055 9745 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:49.432453 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:49.427040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.5.209.126:35665)
0808 02:07:49.427171 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:49.427176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:49.427470 (+ 294us) server_negotiation.cc:407] Connection header received
0808 02:07:49.427633 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:49.427636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:49.427697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:49.427786 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:49.428524 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:49.429161 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:49.430113 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:49.430415 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:49.430526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:49.431441 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:49.431445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:49.431447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:49.431928 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:49.431967 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:49.431986 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:49.432087 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:49.432202 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:07:49.432284 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:07:49.437378 9745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc908ce265d641ab9b94c73d73ab5e84: Generated new TSK 0
I20260808 02:07:49.438135 9745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:07:49.447556 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9654
2026-08-08T02:07:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1593 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9774](info): set up 2 sockets
Aug 08 02:07:49 dist-test-slave-bw88 krb5kdc[9774](info): commencing operation
krb5kdc: starting...
W20260808 02:07:50.740756 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 02:07:50.915681 5957 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.175s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 02:07:51.051383 5957 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.136s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 02:07:51.199744 5957 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.148s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/test-user.keytab.
W20260808 02:07:51.722301 5957 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.522s user 0.000s sys 0.002s
Aug 08 02:07:51 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
W20260808 02:07:52.157222 5957 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.209.126: real 0.409s user 0.000s sys 0.004s
I20260808 02:07:52.157855 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38711
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:38711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:52.278019 9791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:52.278286 9791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:52.278328 9791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:52.282362 9791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:52.282428 9791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:52.282451 9791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:52.282471 9791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:52.282490 9791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:52.288388 9791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:38711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:52.289472 9791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:52.290484 9791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:52.299561 9799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:52.299732 9801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:52.299623 9798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:52.299602 9791 server_base.cc:1034] running on GCE node
I20260808 02:07:52.300276 9791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:52.301463 9791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:52.302654 9791 hybrid_clock.cc:648] HybridClock initialized: now 1786154872302635 us; error 35 us; skew 500 ppm
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:52.310166 9791 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:52.311105 9791 webserver.cc:460] Webserver started at http://127.5.209.126:45801/ using document root <none> and password file <none>
I20260808 02:07:52.311687 9791 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:52.311779 9791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:52.312016 9791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:52.314117 9791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "ae3e84a14afaa5df99387efaab4b00de"
server_key_iv: "ba5a16ac93dff292b06662a559ba71ec"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.314687 9791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "ae3e84a14afaa5df99387efaab4b00de"
server_key_iv: "ba5a16ac93dff292b06662a559ba71ec"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.318493 9791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:07:52.320935 9810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.322228 9791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:52.322381 9791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "ae3e84a14afaa5df99387efaab4b00de"
server_key_iv: "ba5a16ac93dff292b06662a559ba71ec"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.322489 9791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:52.337994 9791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:52.340878 9791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:52.341035 9791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:52.348260 9791 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:38711
I20260808 02:07:52.348618 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:38711 every 8 connection(s)
I20260808 02:07:52.349257 9791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:52.352290 9874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:52.355743 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9791
I20260808 02:07:52.355898 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:52.356279 5957 external_mini_cluster.cc:1428] Setting key 8414ae8b60d08ff5b31254d081612af4
I20260808 02:07:52.358932 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a: Bootstrap starting.
I20260808 02:07:52.361299 9874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:52.362134 9874 log.cc:824] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:52.364707 9874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a: No bootstrap required, opened a new log
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:52.368875 9874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } }
I20260808 02:07:52.369186 9874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:52.369277 9874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de9167f040f4ce7b9134ccfd2fe9a0a, State: Initialized, Role: FOLLOWER
I20260808 02:07:52.369882 9874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } }
I20260808 02:07:52.370137 9874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:52.370244 9874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:52.370376 9874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:52.371626 9874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } }
I20260808 02:07:52.372007 9874 leader_election.cc:302] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5de9167f040f4ce7b9134ccfd2fe9a0a; no voters:
I20260808 02:07:52.372357 9874 leader_election.cc:288] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:52.372416 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:52.357883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.5.209.126:38711)
0808 02:07:52.358352 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:52.358368 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:52.358402 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:07:52.359092 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:52.359113 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:52.359441 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:52.359785 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:52.360642 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.361742 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.362747 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.363421 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.363668 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:52.366558 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:52.366562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:52.366575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:52.366609 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:52.368987 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.369621 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.369630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.369636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.369770 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.370126 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.370131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.370132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.370676 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:52.370837 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:52.371180 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 02:07:52.371482 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260808 02:07:52.372466 9881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:52.372828 9881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [term 1 LEADER]: Becoming Leader. State: Replica: 5de9167f040f4ce7b9134ccfd2fe9a0a, State: Running, Role: LEADER
I20260808 02:07:52.373186 9881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } }
I20260808 02:07:52.373807 9874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:52.374445 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5de9167f040f4ce7b9134ccfd2fe9a0a. Latest consensus state: current_term: 1 leader_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } } }
I20260808 02:07:52.374622 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:52.375083 9888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:52.376233 9882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de9167f040f4ce7b9134ccfd2fe9a0a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38711 } } }
I20260808 02:07:52.376370 9882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:52.378997 9888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:52.384797 9888 catalog_manager.cc:1347] Generated new cluster ID: c513e0fdcc194c339601fdacc94d6c48
I20260808 02:07:52.384867 9888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:52.400745 9888 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:52.402231 9888 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:52.415253 9888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5de9167f040f4ce7b9134ccfd2fe9a0a: Generated new TSK 0
I20260808 02:07:52.415967 9888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:52.457018 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:52.573578 9915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:52.573983 9915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:52.574069 9915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:52.578297 9915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:52.578382 9915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:52.578472 9915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:52.583808 9915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:52.584935 9915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:52.586109 9915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:52.596189 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:52.596204 9923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:52.596167 9928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:52.596729 9915 server_base.cc:1034] running on GCE node
I20260808 02:07:52.597141 9915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:52.597803 9915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:52.598984 9915 hybrid_clock.cc:648] HybridClock initialized: now 1786154872598961 us; error 29 us; skew 500 ppm
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:52.602041 9915 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:52.602900 9915 webserver.cc:460] Webserver started at http://127.5.209.65:40931/ using document root <none> and password file <none>
I20260808 02:07:52.603466 9915 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:52.603531 9915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:52.603718 9915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:52.605811 9915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "7c5c9f10bfc248dabbba4ea3bdb6e2f0"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "2027dd3e84c5820582f363b842239b0e"
server_key_iv: "57611440136727036cd6ac5ae38496c8"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.606331 9915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "7c5c9f10bfc248dabbba4ea3bdb6e2f0"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "2027dd3e84c5820582f363b842239b0e"
server_key_iv: "57611440136727036cd6ac5ae38496c8"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.610126 9915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:07:52.612501 9936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.613793 9915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:52.613940 9915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "7c5c9f10bfc248dabbba4ea3bdb6e2f0"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "2027dd3e84c5820582f363b842239b0e"
server_key_iv: "57611440136727036cd6ac5ae38496c8"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.614063 9915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:52.647254 9915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:52.650873 9915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:52.651096 9915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:52.651773 9915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:52.652665 9915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:52.652740 9915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.652822 9915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:52.652872 9915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.662096 9915 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:37171
I20260808 02:07:52.662129 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:37171 every 8 connection(s)
I20260808 02:07:52.663029 9915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:52.664705 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 9915
I20260808 02:07:52.664857 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:52.665227 5957 external_mini_cluster.cc:1428] Setting key 0a0df714aeefa82fa8d949926809b124
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:52.679273 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:52.664988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:34841 (local address 127.5.209.126:38711)
0808 02:07:52.665186 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:52.665192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:52.666220 (+ 1028us) server_negotiation.cc:407] Connection header received
0808 02:07:52.668271 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:52.668274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:52.668315 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:52.668386 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:52.669940 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.670490 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.671686 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.671924 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.672039 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:52.675049 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:52.675055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:52.675060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:52.675121 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:52.676906 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.677502 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.677505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.677507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.677586 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.677994 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.678029 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.678035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.678263 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:52.678359 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:52.678857 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 02:07:52.678973 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:07:52.680191 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:52.665469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38711 (local address 127.5.209.65:34841)
0808 02:07:52.666047 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:52.666107 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:52.668017 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:52.668573 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:52.668585 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:52.669134 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:52.669751 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:52.669766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.670626 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.670630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:52.671551 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:52.671561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.672082 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.672087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:52.672277 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:52.672917 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:52.672944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:52.674816 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:52.677051 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:52.677059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:52.677084 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:52.677381 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:52.677669 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:52.677672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:52.677705 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:52.677837 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:52.678372 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:52.678378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:52.678618 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 02:07:52.678770 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 02:07:52.679018 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":163,"threads_started":1}
I20260808 02:07:52.681594 10088 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38711
I20260808 02:07:52.681871 10088 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:52.682374 10088 heartbeater.cc:511] Master 127.5.209.126:38711 requested a full tablet report, sending...
I20260808 02:07:52.683881 9834 ts_manager.cc:194] Registered new tserver with Master: 7c5c9f10bfc248dabbba4ea3bdb6e2f0 (127.5.209.65:37171)
I20260808 02:07:52.685045 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:52.697558 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:52.826583 10095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:52.826835 10095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:52.826880 10095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:52.830899 10095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:52.830952 10095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:52.831014 10095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:52.836141 10095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:52.837064 10095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:52.838045 10095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:52.847731 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:52.847720 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:52.847841 10095 server_base.cc:1034] running on GCE node
W20260808 02:07:52.847741 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:52.848388 10095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:52.849023 10095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:52.850205 10095 hybrid_clock.cc:648] HybridClock initialized: now 1786154872850192 us; error 24 us; skew 500 ppm
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:52.852839 10095 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:52.853698 10095 webserver.cc:460] Webserver started at http://127.5.209.66:46385/ using document root <none> and password file <none>
I20260808 02:07:52.854206 10095 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:52.854287 10095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:52.854501 10095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:52.856518 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "99df878ad0514bde891a4e531ea6360f"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "45c7299cf87f252d927c2f949ed250f9"
server_key_iv: "1af5320009970f1a0901d91f93b50ce3"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.857090 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "99df878ad0514bde891a4e531ea6360f"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "45c7299cf87f252d927c2f949ed250f9"
server_key_iv: "1af5320009970f1a0901d91f93b50ce3"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.860797 10095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:52.863354 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.864637 10095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:52.864759 10095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "99df878ad0514bde891a4e531ea6360f"
format_stamp: "Formatted at 2026-08-08 02:07:52 on dist-test-slave-bw88"
server_key: "45c7299cf87f252d927c2f949ed250f9"
server_key_iv: "1af5320009970f1a0901d91f93b50ce3"
server_key_version: "encryptionkey@0"
I20260808 02:07:52.864864 10095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:52.874982 10095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:52.878134 10095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:52.878327 10095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:52.878908 10095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:52.879825 10095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:52.879875 10095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.879938 10095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:52.879972 10095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:52.889519 10095 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:37671
I20260808 02:07:52.889582 10252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:37671 every 8 connection(s)
I20260808 02:07:52.890451 10095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:52.895123 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10095
I20260808 02:07:52.895255 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:52.895524 5957 external_mini_cluster.cc:1428] Setting key 6fed03b6d2550f07b85605beb4f87ad3
Aug 08 02:07:52 dist-test-slave-bw88 krb5kdc[9774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:52.906872 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:52.892422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:37977 (local address 127.5.209.126:38711)
0808 02:07:52.892571 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:52.892575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:52.893393 (+ 818us) server_negotiation.cc:407] Connection header received
0808 02:07:52.895038 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:52.895042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:52.895122 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:52.895221 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:52.896780 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.897568 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.898762 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.899165 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.899293 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:52.902573 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:52.902576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:52.902578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:52.902601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:52.904459 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.905128 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.905131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.905132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.905212 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:52.905676 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:52.905679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:52.905680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:52.905864 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:52.905947 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:52.906639 (+ 692us) server_negotiation.cc:299] Negotiation successful
0808 02:07:52.906749 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:07:52.908011 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:52.892737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38711 (local address 127.5.209.66:37977)
0808 02:07:52.893215 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:52.893259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:52.894796 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:52.895444 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:52.895465 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:52.895912 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:52.896625 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:52.896647 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.897705 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.897708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:52.898634 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:52.898644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:52.899370 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:52.899376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:52.899608 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:52.900244 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:52.900270 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:52.902402 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:52.904577 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:52.904587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:52.904613 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:52.905015 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:52.905314 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:52.905317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:52.905322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:52.905561 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:52.905982 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:52.905989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:52.906309 (+ 320us) client_negotiation.cc:769] Sending connection context
0808 02:07:52.906528 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 02:07:52.906904 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:07:52.909200 10253 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38711
I20260808 02:07:52.909592 10253 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:52.910104 10253 heartbeater.cc:511] Master 127.5.209.126:38711 requested a full tablet report, sending...
I20260808 02:07:52.911204 9834 ts_manager.cc:194] Registered new tserver with Master: 99df878ad0514bde891a4e531ea6360f (127.5.209.66:37671)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:07:52.911851 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:37977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:52.928416 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:53.045068 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:53.045310 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:53.045437 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:53.049437 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:53.049495 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:53.049575 10259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:53.055115 10259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:53.056201 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:53.057261 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:53.067250 10271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:53.067240 10266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:53.067237 10268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:53.067292 10259 server_base.cc:1034] running on GCE node
I20260808 02:07:53.068024 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:53.068681 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:53.069839 10259 hybrid_clock.cc:648] HybridClock initialized: now 1786154873069812 us; error 41 us; skew 500 ppm
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154873, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:53.072533 10259 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:53.073491 10259 webserver.cc:460] Webserver started at http://127.5.209.67:39179/ using document root <none> and password file <none>
I20260808 02:07:53.074074 10259 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:53.074141 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:53.074311 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:53.076464 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "c9c9852e126a471b8b351f83d6e3fbdd"
format_stamp: "Formatted at 2026-08-08 02:07:53 on dist-test-slave-bw88"
server_key: "5723a3f0c134e07bf91cf0d9a3fcb9e3"
server_key_iv: "45a17aed866a78b8e4d27f52ec827343"
server_key_version: "encryptionkey@0"
I20260808 02:07:53.076970 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c9852e126a471b8b351f83d6e3fbdd"
format_stamp: "Formatted at 2026-08-08 02:07:53 on dist-test-slave-bw88"
server_key: "5723a3f0c134e07bf91cf0d9a3fcb9e3"
server_key_iv: "45a17aed866a78b8e4d27f52ec827343"
server_key_version: "encryptionkey@0"
I20260808 02:07:53.080951 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 02:07:53.083588 10278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:53.085079 10259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:53.085211 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "c9c9852e126a471b8b351f83d6e3fbdd"
format_stamp: "Formatted at 2026-08-08 02:07:53 on dist-test-slave-bw88"
server_key: "5723a3f0c134e07bf91cf0d9a3fcb9e3"
server_key_iv: "45a17aed866a78b8e4d27f52ec827343"
server_key_version: "encryptionkey@0"
I20260808 02:07:53.085379 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:53.102639 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:53.105985 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:53.106189 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:53.106791 10259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:53.107640 10259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:53.107707 10259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:53.107777 10259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:53.107824 10259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:53.117146 10259 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:35425
I20260808 02:07:53.117216 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:35425 every 8 connection(s)
I20260808 02:07:53.118059 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:53.125461 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10259
I20260808 02:07:53.125614 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:53.125962 5957 external_mini_cluster.cc:1428] Setting key 7d0989daeb1eca51d336daf389d693c9
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[9774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154873, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:53.131884 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:53.119886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:54961 (local address 127.5.209.126:38711)
0808 02:07:53.120036 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:53.120040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:53.120851 (+ 811us) server_negotiation.cc:407] Connection header received
0808 02:07:53.122334 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:53.122338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:53.122397 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:53.122480 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:53.123630 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.124176 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.125444 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.125875 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.125963 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:53.128747 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:53.128750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:53.128752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:53.128779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:53.130161 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:53.130627 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:53.130630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:53.130631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:53.130706 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:53.130986 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:53.130988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:53.130989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:53.131169 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:53.131245 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:53.131668 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 02:07:53.131766 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:07:53.132676 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:53.120258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38711 (local address 127.5.209.67:54961)
0808 02:07:53.120707 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:53.120750 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:53.122121 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:53.122651 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:53.122657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:53.123000 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:53.123472 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:53.123485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.124310 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.124314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:53.125284 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:53.125299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.126026 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.126031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:53.126225 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:53.126906 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:53.126961 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:53.128592 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:53.130267 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:53.130272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:53.130283 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:53.130529 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:53.130781 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:53.130783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:53.130785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:53.130912 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:53.131255 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:53.131260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:53.131473 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 02:07:53.131630 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 02:07:53.131866 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260808 02:07:53.133847 10424 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38711
I20260808 02:07:53.134073 10424 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:53.134542 10424 heartbeater.cc:511] Master 127.5.209.126:38711 requested a full tablet report, sending...
I20260808 02:07:53.135571 9834 ts_manager.cc:194] Registered new tserver with Master: c9c9852e126a471b8b351f83d6e3fbdd (127.5.209.67:35425)
I20260808 02:07:53.136101 9834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:54961
I20260808 02:07:53.139868 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[9774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[9774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:53.166754 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:53.158248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.5.209.126:38711)
0808 02:07:53.158391 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:53.158395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:53.158577 (+ 182us) server_negotiation.cc:407] Connection header received
0808 02:07:53.158687 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:53.158689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:53.158738 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:53.158806 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:53.159524 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.160115 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.160946 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:53.161208 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:53.161305 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:53.163695 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:53.163698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:53.163700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:53.163723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:53.165221 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:53.165646 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:53.165649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:53.165650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:53.165735 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:53.165984 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:53.165986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:53.165987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:53.166219 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:53.166327 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:53.166528 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 02:07:53.166625 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 02:07:53.167182 9834 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55480
I20260808 02:07:53.168268 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9915
I20260808 02:07:53.188897 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10095
I20260808 02:07:53.202204 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10259
I20260808 02:07:53.212652 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 9791
2026-08-08T02:07:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3765 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[10440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[10440](info): set up 2 sockets
Aug 08 02:07:53 dist-test-slave-bw88 krb5kdc[10440](info): commencing operation
krb5kdc: starting...
W20260808 02:07:54.416611 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:54 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:54.549770 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:36229
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:54.688005 10460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:54.688282 10460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:54.688323 10460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:54.692288 10460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:54.692358 10460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:54.692377 10460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:54.692394 10460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:54.692409 10460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:54.697770 10460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:54.698833 10460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:54.699854 10460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:54.708696 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:54.708729 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:54.708698 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:54.709097 10460 server_base.cc:1034] running on GCE node
I20260808 02:07:54.709702 10460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:54.710768 10460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:54.711958 10460 hybrid_clock.cc:648] HybridClock initialized: now 1786154874711935 us; error 49 us; skew 500 ppm
Aug 08 02:07:54 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:54.714730 10460 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:54.715633 10460 webserver.cc:460] Webserver started at http://127.5.209.126:34113/ using document root <none> and password file <none>
I20260808 02:07:54.716214 10460 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:54.716308 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:54.716544 10460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:54.718642 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "9078573d59b04ce9a166a1caf15ab4d9"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "0c319bbe35b4557a6f525afc0a329625"
server_key_iv: "6c4e49ce4da3e8abb9c049d956f959e2"
server_key_version: "encryptionkey@0"
I20260808 02:07:54.719223 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "9078573d59b04ce9a166a1caf15ab4d9"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "0c319bbe35b4557a6f525afc0a329625"
server_key_iv: "6c4e49ce4da3e8abb9c049d956f959e2"
server_key_version: "encryptionkey@0"
I20260808 02:07:54.722853 10460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 02:07:54.725724 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:54.727123 10460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:54.727259 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9078573d59b04ce9a166a1caf15ab4d9"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "0c319bbe35b4557a6f525afc0a329625"
server_key_iv: "6c4e49ce4da3e8abb9c049d956f959e2"
server_key_version: "encryptionkey@0"
I20260808 02:07:54.727384 10460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:54.741429 10460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:54.744179 10460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:54.744387 10460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:54.752079 10460 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:36229
I20260808 02:07:54.752107 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:36229 every 8 connection(s)
I20260808 02:07:54.753094 10460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:54.756138 10554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:54.758528 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10460
I20260808 02:07:54.758663 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:54.758973 5957 external_mini_cluster.cc:1428] Setting key 261bb1941f9e7f50457870d62018bc0f
I20260808 02:07:54.762805 10554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9: Bootstrap starting.
I20260808 02:07:54.765110 10554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:54.765944 10554 log.cc:824] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:07:54 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:54.768527 10554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9: No bootstrap required, opened a new log
I20260808 02:07:54.772038 10554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } }
I20260808 02:07:54.772289 10554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:54.772343 10554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9078573d59b04ce9a166a1caf15ab4d9, State: Initialized, Role: FOLLOWER
I20260808 02:07:54.772848 10554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } }
I20260808 02:07:54.773016 10554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:54.773125 10554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:54.773253 10554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:54.773705 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:54.760522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.5.209.126:36229)
0808 02:07:54.760993 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:54.761008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:54.761046 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:07:54.761866 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:54.761887 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:54.762235 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:54.762580 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:54.763413 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:54.764401 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:54.765266 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:54.765811 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:54.765995 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:54.768820 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:54.768825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:54.768838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:54.768866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:54.770614 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:54.771073 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:54.771080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:54.771085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:54.771203 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:54.771496 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:54.771499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:54.771501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:54.771897 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:54.772123 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:54.772393 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:07:54.772737 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260808 02:07:54.774415 10554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } }
I20260808 02:07:54.774777 10554 leader_election.cc:302] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9078573d59b04ce9a166a1caf15ab4d9; no voters:
I20260808 02:07:54.775180 10554 leader_election.cc:288] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:54.775254 10560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:54.775466 10560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [term 1 LEADER]: Becoming Leader. State: Replica: 9078573d59b04ce9a166a1caf15ab4d9, State: Running, Role: LEADER
I20260808 02:07:54.775776 10560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } }
I20260808 02:07:54.776518 10554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:54.777015 10560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9078573d59b04ce9a166a1caf15ab4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } } }
I20260808 02:07:54.777133 10560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:54.777637 10561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9078573d59b04ce9a166a1caf15ab4d9. Latest consensus state: current_term: 1 leader_uuid: "9078573d59b04ce9a166a1caf15ab4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9078573d59b04ce9a166a1caf15ab4d9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36229 } } }
I20260808 02:07:54.777796 10561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:54.778828 10565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:54.781756 10565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:54.787110 10565 catalog_manager.cc:1347] Generated new cluster ID: 991dcb5e24434eb9a3100ae8b6e78fff
I20260808 02:07:54.787226 10565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:54.797227 10565 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:54.798172 10565 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:54.811169 10565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9078573d59b04ce9a166a1caf15ab4d9: Generated new TSK 0
I20260808 02:07:54.811851 10565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:54.845078 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:54.962841 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:54.963088 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:54.963156 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:54.967201 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:54.967257 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:54.967368 10586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:54.972470 10586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:54.973515 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:54.974548 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:54.984769 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:54.984769 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:54.984822 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:54.984994 10586 server_base.cc:1034] running on GCE node
I20260808 02:07:54.985512 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:54.986207 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:54.987402 10586 hybrid_clock.cc:648] HybridClock initialized: now 1786154874987387 us; error 26 us; skew 500 ppm
Aug 08 02:07:54 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:54.990240 10586 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:54.991210 10586 webserver.cc:460] Webserver started at http://127.5.209.65:36923/ using document root <none> and password file <none>
I20260808 02:07:54.991854 10586 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:54.991946 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:54.992190 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:54.994334 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "1dca3d7c955b4be1a392e25c03b2934c"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "d274e95254175f7be9279cf4aa3932ba"
server_key_iv: "6ba06a9a37d6fd243296c6fffa2dad2a"
server_key_version: "encryptionkey@0"
I20260808 02:07:54.994927 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "1dca3d7c955b4be1a392e25c03b2934c"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "d274e95254175f7be9279cf4aa3932ba"
server_key_iv: "6ba06a9a37d6fd243296c6fffa2dad2a"
server_key_version: "encryptionkey@0"
I20260808 02:07:54.998726 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:55.001163 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.002583 10586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:55.002743 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "1dca3d7c955b4be1a392e25c03b2934c"
format_stamp: "Formatted at 2026-08-08 02:07:54 on dist-test-slave-bw88"
server_key: "d274e95254175f7be9279cf4aa3932ba"
server_key_iv: "6ba06a9a37d6fd243296c6fffa2dad2a"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.002863 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:55.014935 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:55.017880 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:55.018097 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:55.018725 10586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:55.019600 10586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:55.019675 10586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.019747 10586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:55.019790 10586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.029042 10586 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:43305
I20260808 02:07:55.029074 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:43305 every 8 connection(s)
I20260808 02:07:55.030021 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:55.032588 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10586
I20260808 02:07:55.032733 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:55.033102 5957 external_mini_cluster.cc:1428] Setting key f85ec3787e3d7551c30db6de80131890
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154874, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:55.045815 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.031844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:38081 (local address 127.5.209.126:36229)
0808 02:07:55.031986 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:55.031990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:55.032922 (+ 932us) server_negotiation.cc:407] Connection header received
0808 02:07:55.034542 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:55.034545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:55.034614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:55.034701 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:55.035852 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.036699 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.038512 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.038959 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.039108 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.041930 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:55.041937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:55.041943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:55.041980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:55.043702 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.044207 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.044213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.044215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.044313 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.044654 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.044658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.044660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.044896 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:55.045027 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:55.045565 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 02:07:55.045685 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:07:55.046633 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.032275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36229 (local address 127.5.209.65:38081)
0808 02:07:55.032754 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:55.032807 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:55.034313 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:55.034861 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:55.034867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:55.035227 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:55.035699 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.035712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.036851 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.036856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.038383 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.038396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.039140 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.039143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.039288 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.039912 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:55.039929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:55.041745 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:55.043819 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.043824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.043836 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.044082 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.044411 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.044413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.044415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.044542 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.045021 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:55.045032 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:55.045271 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 02:07:55.045477 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 02:07:55.045743 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260808 02:07:55.047923 10742 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:07:55.048188 10742 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:55.048748 10742 heartbeater.cc:511] Master 127.5.209.126:36229 requested a full tablet report, sending...
I20260808 02:07:55.050323 10507 ts_manager.cc:194] Registered new tserver with Master: 1dca3d7c955b4be1a392e25c03b2934c (127.5.209.65:43305)
I20260808 02:07:55.051474 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:38081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:55.065034 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:55.186766 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:55.187036 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:55.187112 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:55.191256 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:55.191326 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:55.191432 10748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:55.196924 10748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:55.198115 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:55.199235 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:55.209769 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:55.209839 10748 server_base.cc:1034] running on GCE node
W20260808 02:07:55.209781 10755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:55.209791 10754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:55.210593 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:55.211288 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:55.212510 10748 hybrid_clock.cc:648] HybridClock initialized: now 1786154875212496 us; error 35 us; skew 500 ppm
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:55.215484 10748 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:55.216341 10748 webserver.cc:460] Webserver started at http://127.5.209.66:34297/ using document root <none> and password file <none>
I20260808 02:07:55.216894 10748 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:55.216955 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:55.217123 10748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:55.219231 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "cefb8d9b14304c83a9a6d59003371344"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c6d2b397e1ed0b1b5ace598b03b1f702"
server_key_iv: "6264464a20bd79201cbb629362ce7153"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.219810 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "cefb8d9b14304c83a9a6d59003371344"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c6d2b397e1ed0b1b5ace598b03b1f702"
server_key_iv: "6264464a20bd79201cbb629362ce7153"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.223783 10748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:07:55.226549 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.227910 10748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:55.228068 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "cefb8d9b14304c83a9a6d59003371344"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c6d2b397e1ed0b1b5ace598b03b1f702"
server_key_iv: "6264464a20bd79201cbb629362ce7153"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.228188 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:55.239070 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:55.241989 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:55.242192 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:55.242798 10748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:55.243645 10748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:55.243718 10748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.243794 10748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:55.243839 10748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.252920 10748 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:35487
I20260808 02:07:55.252941 10916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:35487 every 8 connection(s)
I20260808 02:07:55.253841 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:55.262393 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10748
I20260808 02:07:55.262545 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:55.262910 5957 external_mini_cluster.cc:1428] Setting key ecf899bdcbc7213170e473a1299bdd28
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:55.268621 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.255695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:33095 (local address 127.5.209.126:36229)
0808 02:07:55.255845 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:55.255848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:55.256719 (+ 871us) server_negotiation.cc:407] Connection header received
0808 02:07:55.258293 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:55.258297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:55.258358 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:55.258441 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:55.259607 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.260159 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.261350 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.261593 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.261711 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.264471 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:55.264475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:55.264478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:55.264507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:55.266065 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.266744 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.266751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.266754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.266863 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.267294 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.267299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.267304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.267613 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:55.267729 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:55.268222 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 02:07:55.268367 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:07:55.269430 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.256070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36229 (local address 127.5.209.66:33095)
0808 02:07:55.256562 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:55.256606 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:55.258067 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:55.258617 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:55.258624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:55.258971 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:55.259443 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.259456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.260300 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.260304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.261202 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.261212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.261754 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.261759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.261935 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.262540 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:55.262564 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:55.264335 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:55.266195 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.266204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.266223 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.266609 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.266967 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.266969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.266971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.267185 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.267730 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:55.267735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:55.267971 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 02:07:55.268146 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:07:55.268422 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260808 02:07:55.270723 10917 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36229
I20260808 02:07:55.271009 10917 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:55.271703 10917 heartbeater.cc:511] Master 127.5.209.126:36229 requested a full tablet report, sending...
I20260808 02:07:55.273028 10507 ts_manager.cc:194] Registered new tserver with Master: cefb8d9b14304c83a9a6d59003371344 (127.5.209.66:35487)
I20260808 02:07:55.273699 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:33095
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:55.296798 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:55.415758 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:55.416008 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:55.416049 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:55.420027 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:55.420080 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:55.420136 10924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:55.425307 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:55.426366 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:55.427352 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:55.437513 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:55.437513 10930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:55.437683 10924 server_base.cc:1034] running on GCE node
W20260808 02:07:55.437855 10931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:55.438268 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:55.438885 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:55.440084 10924 hybrid_clock.cc:648] HybridClock initialized: now 1786154875440070 us; error 26 us; skew 500 ppm
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:55.442930 10924 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:55.443878 10924 webserver.cc:460] Webserver started at http://127.5.209.67:42665/ using document root <none> and password file <none>
I20260808 02:07:55.444453 10924 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:55.444545 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:55.444794 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:55.446890 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "e3914d2fe9bf4787b623f74ce17dcc16"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c7d6a15898a8f0657e6c10144c634fdc"
server_key_iv: "d530a1dced1357a7a6d644b5e1a32fc6"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.447463 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "e3914d2fe9bf4787b623f74ce17dcc16"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c7d6a15898a8f0657e6c10144c634fdc"
server_key_iv: "d530a1dced1357a7a6d644b5e1a32fc6"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.451215 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 02:07:55.453773 10947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.454960 10924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:55.455098 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "e3914d2fe9bf4787b623f74ce17dcc16"
format_stamp: "Formatted at 2026-08-08 02:07:55 on dist-test-slave-bw88"
server_key: "c7d6a15898a8f0657e6c10144c634fdc"
server_key_iv: "d530a1dced1357a7a6d644b5e1a32fc6"
server_key_version: "encryptionkey@0"
I20260808 02:07:55.455210 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:55.467521 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:55.470517 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:55.470721 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:55.471316 10924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:55.472139 10924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:55.472210 10924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.472280 10924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:55.472321 10924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:55.482421 10924 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:34711
I20260808 02:07:55.482450 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:34711 every 8 connection(s)
I20260808 02:07:55.483345 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:55.484203 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 10924
I20260808 02:07:55.484355 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:55.484738 5957 external_mini_cluster.cc:1428] Setting key edfc8b72b282da4f54463a3e664965f6
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:55.497974 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.485491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:41655 (local address 127.5.209.126:36229)
0808 02:07:55.485771 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:55.485776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:55.486713 (+ 937us) server_negotiation.cc:407] Connection header received
0808 02:07:55.488324 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:55.488328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:55.488379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:55.488452 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:55.489766 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.490331 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.491567 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.491806 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.491921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.494520 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:55.494524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:55.494526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:55.494556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:55.495777 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.496301 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.496305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.496306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.496394 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.496750 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.496753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.496754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.496954 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:55.497074 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:55.497667 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 02:07:55.497787 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260808 02:07:55.498844 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.485957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36229 (local address 127.5.209.67:41655)
0808 02:07:55.486558 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:55.486603 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:55.488073 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:55.488634 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:55.488642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:55.489014 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:55.489615 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.489632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.490472 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.490476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.491423 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:55.491434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.491963 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.491968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:55.492154 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.492726 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:55.492743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:55.494348 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:55.495901 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.495907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.495922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.496185 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.496494 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:55.496497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:55.496499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:55.496641 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:55.497075 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:55.497083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:55.497389 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 02:07:55.497606 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 02:07:55.497922 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260808 02:07:55.499939 11102 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36229
I20260808 02:07:55.500213 11102 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:55.500728 11102 heartbeater.cc:511] Master 127.5.209.126:36229 requested a full tablet report, sending...
I20260808 02:07:55.501852 10507 ts_manager.cc:194] Registered new tserver with Master: e3914d2fe9bf4787b623f74ce17dcc16 (127.5.209.67:34711)
I20260808 02:07:55.502411 10507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:41655
I20260808 02:07:55.510705 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:07:55.539062 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.529335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.5.209.65:43305)
0808 02:07:55.529688 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:55.529703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:55.529730 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:07:55.529784 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:55.529790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:55.529988 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:55.530120 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:55.530880 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.531801 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.532736 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.533185 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.533308 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.535753 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:55.535763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:55.535777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:55.535802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:55.537288 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.537796 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.537802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.537803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.537884 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.538120 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.538122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.538123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.538373 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:55.538543 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:55.538777 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 02:07:55.538906 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[10440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:55.550211 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:55.542160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.5.209.126:36229)
0808 02:07:55.542309 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:55.542312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:55.542327 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:07:55.542411 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:55.542413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:55.542455 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:55.542527 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:55.543071 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.543574 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.544300 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:55.544512 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:55.544587 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:55.547149 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:55.547152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:55.547154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:55.547180 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:55.548623 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.548986 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.548989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.548990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.549076 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:55.549302 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:55.549304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:55.549305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:55.549569 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:55.549681 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:55.549881 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 02:07:55.550015 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 02:07:55.550657 10507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44136
I20260808 02:07:55.551311 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10586
I20260808 02:07:55.565022 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10748
I20260808 02:07:55.581730 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10924
I20260808 02:07:55.592262 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 10460
2026-08-08T02:07:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2376 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[11120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[11120](info): set up 2 sockets
Aug 08 02:07:55 dist-test-slave-bw88 krb5kdc[11120](info): commencing operation
krb5kdc: starting...
W20260808 02:07:56.842811 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:07:56 dist-test-slave-bw88 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:07:56.972896 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43537
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:57.089104 11139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:57.089546 11139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:57.089635 11139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:57.096387 11139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:57.096470 11139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:57.096519 11139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:57.096562 11139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:57.096596 11139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:57.105046 11139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:43537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:57.106256 11139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:57.107348 11139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:57.116101 11145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.116073 11149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.116061 11146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:57.116371 11139 server_base.cc:1034] running on GCE node
I20260808 02:07:57.116883 11139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:57.118027 11139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:57.119230 11139 hybrid_clock.cc:648] HybridClock initialized: now 1786154877119209 us; error 32 us; skew 500 ppm
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:57.122061 11139 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:07:57.122836 11139 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:57.122932 11139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:57.123172 11139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:57.125234 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "794540fe13e54f64a532874adb58b069"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "2b7a07fa589a736ca7b9c618994c5540"
server_key_iv: "99cdf4badc0e688aea1e0d8597493e8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.125873 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "794540fe13e54f64a532874adb58b069"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "2b7a07fa589a736ca7b9c618994c5540"
server_key_iv: "99cdf4badc0e688aea1e0d8597493e8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.129639 11139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:57.132076 11156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.133536 11139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:57.133672 11139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "794540fe13e54f64a532874adb58b069"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "2b7a07fa589a736ca7b9c618994c5540"
server_key_iv: "99cdf4badc0e688aea1e0d8597493e8c"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.133790 11139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:57.162568 11139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:57.165741 11139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:57.165958 11139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:57.173595 11139 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:43537
I20260808 02:07:57.173940 11221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:43537 every 8 connection(s)
I20260808 02:07:57.174338 11139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:57.177316 11222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:57.180342 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11139
I20260808 02:07:57.180533 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:57.180819 5957 external_mini_cluster.cc:1428] Setting key 01502dd072b059468d93ec32b3667f6a
I20260808 02:07:57.183712 11222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: Bootstrap starting.
I20260808 02:07:57.186594 11222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:57.187551 11222 log.cc:824] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:57.190214 11222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: No bootstrap required, opened a new log
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:57.195168 11222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } }
I20260808 02:07:57.195447 11222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:57.195508 11222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794540fe13e54f64a532874adb58b069, State: Initialized, Role: FOLLOWER
I20260808 02:07:57.195987 11222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } }
I20260808 02:07:57.196154 11222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:57.196221 11222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:57.196317 11222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:57.196419 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.182281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.5.209.126:43537)
0808 02:07:57.182722 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.182738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.182775 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:07:57.183415 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.183433 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.183763 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.184101 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:57.185096 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.186187 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.187463 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.187984 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.188190 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.191064 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:57.191070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:57.191085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:57.191117 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:57.193190 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.193729 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.193736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.193745 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.193890 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.194162 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.194165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.194167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.194717 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:57.194928 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.195130 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.195438 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":146,"threads_started":1}
I20260808 02:07:57.197263 11222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } }
I20260808 02:07:57.197628 11222 leader_election.cc:302] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794540fe13e54f64a532874adb58b069; no voters:
I20260808 02:07:57.197981 11222 leader_election.cc:288] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:57.198089 11229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:57.198320 11229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [term 1 LEADER]: Becoming Leader. State: Replica: 794540fe13e54f64a532874adb58b069, State: Running, Role: LEADER
I20260808 02:07:57.198632 11229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } }
I20260808 02:07:57.199335 11222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:57.200448 11231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794540fe13e54f64a532874adb58b069. Latest consensus state: current_term: 1 leader_uuid: "794540fe13e54f64a532874adb58b069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } } }
I20260808 02:07:57.200593 11231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:57.200456 11230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "794540fe13e54f64a532874adb58b069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794540fe13e54f64a532874adb58b069" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43537 } } }
I20260808 02:07:57.200933 11230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:57.200997 11241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:57.204037 11241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:57.209834 11241 catalog_manager.cc:1347] Generated new cluster ID: 10e1698f6bfb4e82ab45bba4e73b1be1
I20260808 02:07:57.209931 11241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:57.220777 11241 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:57.222057 11241 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:57.239054 11241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: Generated new TSK 0
I20260808 02:07:57.239871 11241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:07:57.280311 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:57.405936 11253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:57.406183 11253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:57.406225 11253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:57.410110 11253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:57.410161 11253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:57.410218 11253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:57.415189 11253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:57.416138 11253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:57.417151 11253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:57.426837 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.426882 11260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.426833 11259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:57.427317 11253 server_base.cc:1034] running on GCE node
I20260808 02:07:57.427748 11253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:57.428370 11253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:57.429539 11253 hybrid_clock.cc:648] HybridClock initialized: now 1786154877429509 us; error 39 us; skew 500 ppm
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:57.436297 11253 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:07:57.437176 11253 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:57.437268 11253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:57.437525 11253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:57.439706 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "729620b288364cf1aa60a8532f0f6739"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "39c5c654f05cf4cf8e84a37ee9dd99b4"
server_key_iv: "017267a000b57e9c5522aee711287c65"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.440280 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "729620b288364cf1aa60a8532f0f6739"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "39c5c654f05cf4cf8e84a37ee9dd99b4"
server_key_iv: "017267a000b57e9c5522aee711287c65"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.444073 11253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 02:07:57.446653 11270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.447894 11253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:57.448037 11253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "729620b288364cf1aa60a8532f0f6739"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "39c5c654f05cf4cf8e84a37ee9dd99b4"
server_key_iv: "017267a000b57e9c5522aee711287c65"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.448149 11253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:57.467758 11253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:57.471046 11253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:57.471274 11253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:57.471890 11253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:57.472780 11253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:57.472852 11253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.472923 11253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:57.472963 11253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.482261 11253 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:37715
I20260808 02:07:57.482297 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:37715 every 8 connection(s)
I20260808 02:07:57.482894 11253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:57.489061 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11253
I20260808 02:07:57.489215 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:57.489596 5957 external_mini_cluster.cc:1428] Setting key 13efec7eda76dee5a4ae8954c3f7b39e
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:57.497877 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.484808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:32925 (local address 127.5.209.126:43537)
0808 02:07:57.484983 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.484987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.485886 (+ 899us) server_negotiation.cc:407] Connection header received
0808 02:07:57.487193 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.487196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.487246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.487321 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:57.488547 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.489079 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.490554 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.490805 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.490923 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.493830 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:57.493836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:57.493841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:57.493877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:57.495469 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.496173 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.496176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.496177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.496255 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.496709 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.496711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.496713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.496891 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:57.497003 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.497568 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.497701 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:07:57.498745 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.485203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43537 (local address 127.5.209.65:32925)
0808 02:07:57.485726 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.485771 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.486978 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.487549 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.487556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.487932 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:57.488414 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.488428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.489229 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.489235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.490431 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.490441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.490964 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.490968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.491172 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.491801 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:57.491827 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:57.493644 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:57.495605 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.495614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.495633 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.496042 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.496352 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.496358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.496363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.496595 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.497008 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:57.497017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:57.497277 (+ 260us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.497506 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.497855 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":151,"threads_started":1}
I20260808 02:07:57.500083 11420 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43537
I20260808 02:07:57.500404 11420 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:57.501029 11420 heartbeater.cc:511] Master 127.5.209.126:43537 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:07:57.502820 11179 ts_manager.cc:194] Registered new tserver with Master: 729620b288364cf1aa60a8532f0f6739 (127.5.209.65:37715)
I20260808 02:07:57.504098 11179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:32925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:07:57.520622 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:57.638720 11428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:57.638991 11428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:57.639070 11428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:57.643119 11428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:57.643178 11428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:57.643276 11428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:57.648679 11428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:57.649843 11428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:57.650954 11428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:57.660693 11433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.660744 11438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.660729 11435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:57.660893 11428 server_base.cc:1034] running on GCE node
I20260808 02:07:57.661511 11428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:57.662184 11428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:57.663385 11428 hybrid_clock.cc:648] HybridClock initialized: now 1786154877663360 us; error 47 us; skew 500 ppm
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:57.666366 11428 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:07:57.667187 11428 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:57.667284 11428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:57.667534 11428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:57.669658 11428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "8a2447f3cd6b4790826d51922c99ba5f"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "ba0fa0a947347570030391c7286ec204"
server_key_iv: "16014060d4a3f8b79998fb0834dd09f8"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.670249 11428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "8a2447f3cd6b4790826d51922c99ba5f"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "ba0fa0a947347570030391c7286ec204"
server_key_iv: "16014060d4a3f8b79998fb0834dd09f8"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.674427 11428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 02:07:57.677083 11446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.678534 11428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:07:57.678711 11428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "8a2447f3cd6b4790826d51922c99ba5f"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "ba0fa0a947347570030391c7286ec204"
server_key_iv: "16014060d4a3f8b79998fb0834dd09f8"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.678838 11428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:57.693243 11428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:57.696349 11428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:57.696552 11428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:57.697170 11428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:57.698103 11428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:57.698153 11428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.698215 11428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:57.698248 11428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.708151 11428 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:40053
I20260808 02:07:57.708233 11607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:40053 every 8 connection(s)
I20260808 02:07:57.708807 11428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:57.717674 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11428
I20260808 02:07:57.717788 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:57.718079 5957 external_mini_cluster.cc:1428] Setting key 90258a836d1e5f5a2929bbed0244e82e
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:57.723279 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.710597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:46397 (local address 127.5.209.126:43537)
0808 02:07:57.710741 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.710745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.711615 (+ 870us) server_negotiation.cc:407] Connection header received
0808 02:07:57.712958 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.712961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.713037 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.713142 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:57.714383 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.714920 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.716086 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.716361 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.716487 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.719235 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:57.719245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:57.719248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:57.719268 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:57.720751 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.721434 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.721437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.721438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.721511 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.721948 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.721954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.721957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.722244 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:57.722356 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.722875 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.723016 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:07:57.723834 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.710991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43537 (local address 127.5.209.66:46397)
0808 02:07:57.711459 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.711511 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.712742 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.713326 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.713377 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.713753 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:57.714236 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.714250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.715058 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.715061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.715952 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.715963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.716539 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.716544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.716716 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.717381 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:57.717409 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:57.719074 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:57.720873 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.720882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.720900 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.721294 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.721606 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.721611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.721616 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.721819 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.722357 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:57.722362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:57.722620 (+ 258us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.722768 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.723009 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260808 02:07:57.724932 11609 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43537
I20260808 02:07:57.725209 11609 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:57.725689 11609 heartbeater.cc:511] Master 127.5.209.126:43537 requested a full tablet report, sending...
I20260808 02:07:57.726706 11179 ts_manager.cc:194] Registered new tserver with Master: 8a2447f3cd6b4790826d51922c99ba5f (127.5.209.66:40053)
I20260808 02:07:57.727244 11179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:46397
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:07:57.748829 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:07:57.867377 11616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:57.867669 11616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:57.867756 11616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:57.871866 11616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:57.871941 11616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:57.872028 11616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:57.877260 11616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:57.878413 11616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:57.879470 11616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:57.889408 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:57.889518 11616 server_base.cc:1034] running on GCE node
W20260808 02:07:57.889413 11624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:57.889413 11626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:57.890089 11616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:57.890795 11616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:57.891995 11616 hybrid_clock.cc:648] HybridClock initialized: now 1786154877891969 us; error 42 us; skew 500 ppm
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:07:57.894936 11616 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:07:57.895632 11616 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:57.895694 11616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:57.895859 11616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:57.897910 11616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "5356430ffbb64bc196e96c187ec269d1"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "bbc25a752d22a2e92cd396d222c89164"
server_key_iv: "1bb3876128529f8ada1da7cc25aec10b"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.898433 11616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "5356430ffbb64bc196e96c187ec269d1"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "bbc25a752d22a2e92cd396d222c89164"
server_key_iv: "1bb3876128529f8ada1da7cc25aec10b"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.902297 11616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:57.904564 11635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.905808 11616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:57.905927 11616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "5356430ffbb64bc196e96c187ec269d1"
format_stamp: "Formatted at 2026-08-08 02:07:57 on dist-test-slave-bw88"
server_key: "bbc25a752d22a2e92cd396d222c89164"
server_key_iv: "1bb3876128529f8ada1da7cc25aec10b"
server_key_version: "encryptionkey@0"
I20260808 02:07:57.906029 11616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:57.920133 11616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:57.922847 11616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:57.923040 11616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:57.923619 11616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:57.924428 11616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:57.924475 11616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.924543 11616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:57.924587 11616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:57.933705 11616 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:37957
I20260808 02:07:57.933724 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:37957 every 8 connection(s)
I20260808 02:07:57.934360 11616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:57.936074 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11616
I20260808 02:07:57.936228 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:57.936542 5957 external_mini_cluster.cc:1428] Setting key 91e8705f070888c306f9bcf808e2bb4e
Aug 08 02:07:57 dist-test-slave-bw88 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154877, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:07:57.948170 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.936109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59527 (local address 127.5.209.126:43537)
0808 02:07:57.936253 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.936257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.937306 (+ 1049us) server_negotiation.cc:407] Connection header received
0808 02:07:57.939117 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.939121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.939178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.939283 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:57.940455 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.940988 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.942241 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.942468 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.942577 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.945005 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:57.945008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:57.945010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:57.945035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:57.946360 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.946838 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.946840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.946841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.946916 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.947226 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.947228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.947230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.947400 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:57.947473 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.947946 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.948042 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:07:57.948973 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.936548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43537 (local address 127.5.209.67:59527)
0808 02:07:57.937151 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.937209 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.938881 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.939469 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.939475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.939840 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:57.940315 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.940329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.941116 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.941119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.942113 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.942124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.942628 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.942633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.942807 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.943346 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:07:57.943363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:57.944839 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:57.946467 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.946471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.946484 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.946750 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.946995 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:07:57.946997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:07:57.946999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:07:57.947145 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:07:57.947487 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:57.947492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:57.947740 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.947898 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.948148 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":143,"threads_started":1}
I20260808 02:07:57.950241 11790 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43537
I20260808 02:07:57.950502 11790 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:57.951025 11790 heartbeater.cc:511] Master 127.5.209.126:43537 requested a full tablet report, sending...
I20260808 02:07:57.952070 11177 ts_manager.cc:194] Registered new tserver with Master: 5356430ffbb64bc196e96c187ec269d1 (127.5.209.67:37957)
I20260808 02:07:57.952678 11177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:59527
I20260808 02:07:57.962615 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:07:57.972062 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.964528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.5.209.126:43537)
0808 02:07:57.964717 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.964721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.964825 (+ 104us) server_negotiation.cc:407] Connection header received
0808 02:07:57.964926 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.964928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.964968 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.965033 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:57.965786 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.966290 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.967179 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.967391 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.967474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.968934 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:57.968937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:57.968939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:07:57.968963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:57.970445 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.970872 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.970875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.970876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.970951 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:07:57.971231 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:07:57.971233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:07:57.971234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:07:57.971437 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:57.971539 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.971754 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.971846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:07:57.975024 11177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:57.978199 11177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:57.993719 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.987231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:37957 (local address 127.0.0.1:46514)
0808 02:07:57.987720 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.987770 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.987907 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.988843 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.988852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.988879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:57.989035 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.989045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990824 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.990827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.992163 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.992174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.993235 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.993239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.993421 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.993444 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.993534 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.993606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":75,"thread_start_us":62,"threads_started":1}
I20260808 02:07:57.993697 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.987915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:37715 (local address 127.0.0.1:39440)
0808 02:07:57.988159 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.988174 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.988260 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.989161 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.989165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.989184 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:57.989297 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.989304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990854 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.990858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.992082 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.992092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.993235 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.993239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.993373 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.993396 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.993449 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.993531 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":37,"thread_start_us":119,"threads_started":1}
I20260808 02:07:57.993682 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.987390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:40053 (local address 127.0.0.1:37496)
0808 02:07:57.988021 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:57.988035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:57.988124 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:57.989406 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:57.989409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:57.989427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:57.989511 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.989515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990569 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.990573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.992169 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:57.992175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.993017 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.993021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:57.993140 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.993178 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:07:57.993322 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 02:07:57.993466 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":67,"thread_start_us":118,"threads_started":1}
I20260808 02:07:57.993848 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.988328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.5.209.66:40053)
0808 02:07:57.988825 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.988836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.988856 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:07:57.988923 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.988927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.989088 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.989214 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:57.989632 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990422 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.992314 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.992884 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.993026 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.993196 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.993556 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.993678 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":134,"threads_started":1}
I20260808 02:07:57.994424 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.987398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.5.209.67:37957)
0808 02:07:57.988268 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.988281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.988325 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:07:57.988409 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.988418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.988642 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.989234 (+ 592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:57.989430 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990669 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.992278 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.993083 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.993287 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.993465 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.994163 (+ 698us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.994268 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":160,"threads_started":1}
I20260808 02:07:57.994693 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:57.988377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.5.209.65:37715)
0808 02:07:57.988671 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:57.988679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:57.988707 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:07:57.988787 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:57.988792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:57.989010 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:57.989190 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:57.989469 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.990667 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.992232 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:57.993083 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:57.994046 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:57.994225 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:57.994269 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 02:07:57.994538 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260808 02:07:57.996968 11518 tablet_service.cc:1467] Processing CreateTablet for tablet fc27da84383f4b9d8297756f6d04a86e (DEFAULT_TABLE table=test-table [id=9449616cce1a47a4a79e6a3c7a7f23a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:57.997239 11703 tablet_service.cc:1467] Processing CreateTablet for tablet fc27da84383f4b9d8297756f6d04a86e (DEFAULT_TABLE table=test-table [id=9449616cce1a47a4a79e6a3c7a7f23a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:57.997424 11335 tablet_service.cc:1467] Processing CreateTablet for tablet fc27da84383f4b9d8297756f6d04a86e (DEFAULT_TABLE table=test-table [id=9449616cce1a47a4a79e6a3c7a7f23a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:57.998075 11518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc27da84383f4b9d8297756f6d04a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:57.998313 11703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc27da84383f4b9d8297756f6d04a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:57.998306 11335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc27da84383f4b9d8297756f6d04a86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:58.004772 11809 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Bootstrap starting.
I20260808 02:07:58.006328 11814 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Bootstrap starting.
I20260808 02:07:58.006470 11811 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: Bootstrap starting.
I20260808 02:07:58.007115 11809 tablet_bootstrap.cc:654] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:58.008298 11809 log.cc:824] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:58.008561 11814 tablet_bootstrap.cc:654] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:58.008561 11811 tablet_bootstrap.cc:654] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:58.009506 11811 log.cc:824] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:58.009935 11814 log.cc:824] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:58.011232 11809 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: No bootstrap required, opened a new log
I20260808 02:07:58.011442 11809 ts_tablet_manager.cc:1397] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 02:07:58.011782 11811 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: No bootstrap required, opened a new log
I20260808 02:07:58.011981 11811 ts_tablet_manager.cc:1397] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 02:07:58.012229 11814 tablet_bootstrap.cc:492] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: No bootstrap required, opened a new log
I20260808 02:07:58.012413 11814 ts_tablet_manager.cc:1397] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 02:07:58.015427 11811 raft_consensus.cc:348] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.015674 11811 raft_consensus.cc:374] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:58.015740 11811 raft_consensus.cc:729] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729620b288364cf1aa60a8532f0f6739, State: Initialized, Role: FOLLOWER
I20260808 02:07:58.016213 11811 consensus_queue.cc:260] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.016291 11814 raft_consensus.cc:348] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.016599 11814 raft_consensus.cc:374] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:58.016427 11809 raft_consensus.cc:348] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.016665 11814 raft_consensus.cc:729] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5356430ffbb64bc196e96c187ec269d1, State: Initialized, Role: FOLLOWER
I20260808 02:07:58.016676 11809 raft_consensus.cc:374] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:58.016727 11809 raft_consensus.cc:729] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2447f3cd6b4790826d51922c99ba5f, State: Initialized, Role: FOLLOWER
I20260808 02:07:58.017138 11420 heartbeater.cc:503] Master 127.5.209.126:43537 was elected leader, sending a full tablet report...
I20260808 02:07:58.017148 11809 consensus_queue.cc:260] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.017148 11814 consensus_queue.cc:260] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.017364 11811 ts_tablet_manager.cc:1428] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260808 02:07:58.018122 11814 ts_tablet_manager.cc:1428] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:58.018113 11609 heartbeater.cc:503] Master 127.5.209.126:43537 was elected leader, sending a full tablet report...
I20260808 02:07:58.018122 11809 ts_tablet_manager.cc:1428] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 02:07:58.019130 11790 heartbeater.cc:503] Master 127.5.209.126:43537 was elected leader, sending a full tablet report...
I20260808 02:07:58.060494 11819 raft_consensus.cc:482] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:58.060755 11819 raft_consensus.cc:504] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.062098 11819 leader_election.cc:288] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a2447f3cd6b4790826d51922c99ba5f (127.5.209.66:40053), 729620b288364cf1aa60a8532f0f6739 (127.5.209.65:37715)
I20260808 02:07:58.066152 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.062290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:40053 (local address 127.5.209.67:59255)
0808 02:07:58.062453 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.062471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:58.062591 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:58.062974 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:58.062983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:58.063026 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:58.063176 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:58.063183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.064149 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.064153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:58.065021 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:58.065031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.065809 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.065811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:58.065893 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:58.065914 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 02:07:58.065975 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:07:58.066030 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":4992}
I20260808 02:07:58.066195 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.062559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59255 (local address 127.5.209.66:40053)
0808 02:07:58.062698 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.062705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.062722 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:07:58.062783 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.062786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.062834 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.062914 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:58.063334 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.064013 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.065163 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.065693 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.065817 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:58.065872 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.065990 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.066056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:07:58.066834 11546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc27da84383f4b9d8297756f6d04a86e" candidate_uuid: "5356430ffbb64bc196e96c187ec269d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2447f3cd6b4790826d51922c99ba5f" is_pre_election: true
I20260808 02:07:58.066975 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.062579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:51953 (local address 127.5.209.65:37715)
0808 02:07:58.062733 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.062738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.063021 (+ 283us) server_negotiation.cc:407] Connection header received
0808 02:07:58.063167 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.063170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.063226 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.063309 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:07:58.063595 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.064195 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.065704 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.066294 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.066423 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:58.066474 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.066795 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.066870 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:07:58.066985 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.062441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:37715 (local address 127.5.209.67:51953)
0808 02:07:58.062926 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.062941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:58.063057 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:58.063333 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:58.063337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:58.063355 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:07:58.063484 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:58.063490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.064312 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.064318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:58.065556 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:58.065572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.066538 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.066542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:58.066660 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:58.066679 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:07:58.066775 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:07:58.066840 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"spinlock_wait_cycles":3840,"thread_start_us":171,"threads_started":1}
I20260808 02:07:58.067204 11546 raft_consensus.cc:2454] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5356430ffbb64bc196e96c187ec269d1 in term 0.
I20260808 02:07:58.067590 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc27da84383f4b9d8297756f6d04a86e" candidate_uuid: "5356430ffbb64bc196e96c187ec269d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729620b288364cf1aa60a8532f0f6739" is_pre_election: true
I20260808 02:07:58.067726 11637 leader_election.cc:302] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5356430ffbb64bc196e96c187ec269d1, 8a2447f3cd6b4790826d51922c99ba5f; no voters:
I20260808 02:07:58.067914 11363 raft_consensus.cc:2454] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5356430ffbb64bc196e96c187ec269d1 in term 0.
I20260808 02:07:58.068068 11819 raft_consensus.cc:2781] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:58.068161 11819 raft_consensus.cc:482] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:58.068213 11819 raft_consensus.cc:3037] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:58.069458 11819 raft_consensus.cc:504] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.069789 11819 leader_election.cc:288] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [CANDIDATE]: Term 1 election: Requested vote from peers 8a2447f3cd6b4790826d51922c99ba5f (127.5.209.66:40053), 729620b288364cf1aa60a8532f0f6739 (127.5.209.65:37715)
I20260808 02:07:58.070147 11546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc27da84383f4b9d8297756f6d04a86e" candidate_uuid: "5356430ffbb64bc196e96c187ec269d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2447f3cd6b4790826d51922c99ba5f"
I20260808 02:07:58.070189 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc27da84383f4b9d8297756f6d04a86e" candidate_uuid: "5356430ffbb64bc196e96c187ec269d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729620b288364cf1aa60a8532f0f6739"
I20260808 02:07:58.070255 11546 raft_consensus.cc:3037] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:58.070340 11363 raft_consensus.cc:3037] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:58.071398 11546 raft_consensus.cc:2454] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5356430ffbb64bc196e96c187ec269d1 in term 1.
I20260808 02:07:58.071491 11363 raft_consensus.cc:2454] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5356430ffbb64bc196e96c187ec269d1 in term 1.
I20260808 02:07:58.071740 11637 leader_election.cc:302] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5356430ffbb64bc196e96c187ec269d1, 8a2447f3cd6b4790826d51922c99ba5f; no voters:
I20260808 02:07:58.071985 11819 raft_consensus.cc:2781] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:58.072250 11819 raft_consensus.cc:686] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 1 LEADER]: Becoming Leader. State: Replica: 5356430ffbb64bc196e96c187ec269d1, State: Running, Role: LEADER
I20260808 02:07:58.072611 11819 consensus_queue.cc:237] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } }
I20260808 02:07:58.075925 11177 catalog_manager.cc:5495] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5356430ffbb64bc196e96c187ec269d1 (127.5.209.67). New cstate: current_term: 1 leader_uuid: "5356430ffbb64bc196e96c187ec269d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5356430ffbb64bc196e96c187ec269d1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:58.097214 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.093519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.5.209.67:37957)
0808 02:07:58.093678 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.093684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.093701 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:07:58.093838 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.093841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.093898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.094015 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:07:58.094291 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.094819 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.095699 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:58.095922 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:58.096036 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:58.096362 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:07:58.096804 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:07:58.096948 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.096986 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.097046 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:07:58.103920 11546 raft_consensus.cc:1261] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 1 FOLLOWER]: Refusing update from remote peer 5356430ffbb64bc196e96c187ec269d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:58.103917 11363 raft_consensus.cc:1261] T fc27da84383f4b9d8297756f6d04a86e P 729620b288364cf1aa60a8532f0f6739 [term 1 FOLLOWER]: Refusing update from remote peer 5356430ffbb64bc196e96c187ec269d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:58.104677 11823 consensus_queue.cc:1035] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729620b288364cf1aa60a8532f0f6739" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:58.104857 11819 consensus_queue.cc:1035] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2447f3cd6b4790826d51922c99ba5f" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:58.112864 11829 mvcc.cc:205] Tried to move back new op lower bound from 7316090380705714176 to 7316090380588109824. Current Snapshot: MvccSnapshot[applied={T|T < 7316090380705714176}]
I20260808 02:07:58.113042 11827 mvcc.cc:205] Tried to move back new op lower bound from 7316090380705714176 to 7316090380588109824. Current Snapshot: MvccSnapshot[applied={T|T < 7316090380705714176}]
I20260808 02:07:58.123960 11177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:58.124162 11177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:58.126796 11177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794540fe13e54f64a532874adb58b069: Sending DeleteTablet for 3 replicas of tablet fc27da84383f4b9d8297756f6d04a86e
I20260808 02:07:58.127671 11518 tablet_service.cc:1514] Processing DeleteTablet for tablet fc27da84383f4b9d8297756f6d04a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:58 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:37496
I20260808 02:07:58.127674 11335 tablet_service.cc:1514] Processing DeleteTablet for tablet fc27da84383f4b9d8297756f6d04a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:58 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:39440
I20260808 02:07:58.128214 11703 tablet_service.cc:1514] Processing DeleteTablet for tablet fc27da84383f4b9d8297756f6d04a86e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:58 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:46514
I20260808 02:07:58.128556 11843 tablet_replica.cc:330] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: stopping tablet replica
I20260808 02:07:58.128623 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11253
I20260808 02:07:58.128958 11843 raft_consensus.cc:2229] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:58.129369 11843 raft_consensus.cc:2258] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:58.132762 11843 ts_tablet_manager.cc:1905] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:58.134174 11844 tablet_replica.cc:330] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: stopping tablet replica
I20260808 02:07:58.134637 11844 raft_consensus.cc:2229] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:58.135274 11844 raft_consensus.cc:2258] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:58.135881 11843 ts_tablet_manager.cc:1918] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:58.136013 11843 log.cc:1196] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/fc27da84383f4b9d8297756f6d04a86e
I20260808 02:07:58.136384 11843 ts_tablet_manager.cc:1939] T fc27da84383f4b9d8297756f6d04a86e P 8a2447f3cd6b4790826d51922c99ba5f: Deleting consensus metadata
I20260808 02:07:58.139545 11844 ts_tablet_manager.cc:1905] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:58.142022 11844 ts_tablet_manager.cc:1918] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:58.142141 11844 log.cc:1196] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/fc27da84383f4b9d8297756f6d04a86e
I20260808 02:07:58.142489 11844 ts_tablet_manager.cc:1939] T fc27da84383f4b9d8297756f6d04a86e P 5356430ffbb64bc196e96c187ec269d1: Deleting consensus metadata
I20260808 02:07:58.143253 11158 catalog_manager.cc:4841] TS 8a2447f3cd6b4790826d51922c99ba5f (127.5.209.66:40053): tablet fc27da84383f4b9d8297756f6d04a86e (table test-table [id=9449616cce1a47a4a79e6a3c7a7f23a6]) successfully deleted
I20260808 02:07:58.143584 11161 catalog_manager.cc:4841] TS 5356430ffbb64bc196e96c187ec269d1 (127.5.209.67:37957): tablet fc27da84383f4b9d8297756f6d04a86e (table test-table [id=9449616cce1a47a4a79e6a3c7a7f23a6]) successfully deleted
W20260808 02:07:58.147589 11161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:37715 (error 108)
I20260808 02:07:58.147742 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11428
I20260808 02:07:58.151593 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.150966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:37715 (local address 127.0.0.1:39442)
0808 02:07:58.151400 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.151493 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
W20260808 02:07:58.152297 11161 catalog_manager.cc:4584] TS 729620b288364cf1aa60a8532f0f6739 (127.5.209.65:37715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc27da84383f4b9d8297756f6d04a86e: Network error: Client connection negotiation failed: client connection to 127.5.209.65:37715: connect: Connection refused (error 111)
I20260808 02:07:58.160171 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11616
I20260808 02:07:58.170519 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11139
2026-08-08T02:07:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2579 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T02:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:58Z Disabled control of system clock
I20260808 02:07:58.206635 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34481
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:07:58.320962 11853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:58.321216 11853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:58.321251 11853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:58.325152 11853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:58.325205 11853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:58.325220 11853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:58.325237 11853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:58.325253 11853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:07:58.329857 11853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:58.330825 11853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:58.332120 11853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:58.340473 11859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:58.340451 11858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:58.340471 11861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.340840 11853 server_base.cc:1034] running on GCE node
I20260808 02:07:58.341496 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:58.342569 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:58.343755 11853 hybrid_clock.cc:648] HybridClock initialized: now 1786154878343733 us; error 31 us; skew 500 ppm
I20260808 02:07:58.345659 11853 webserver.cc:460] Webserver started at http://127.5.209.126:45983/ using document root <none> and password file <none>
I20260808 02:07:58.346279 11853 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:58.346343 11853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:58.346592 11853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:58.348634 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "619b66d8e74f4b1abc285b03311a2b80"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "80146bc8fe42ef3ab6ce019627864c6e"
server_key_iv: "0bc1ffcb845dcbdcbe0c206283ada2a0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.349210 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "619b66d8e74f4b1abc285b03311a2b80"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "80146bc8fe42ef3ab6ce019627864c6e"
server_key_iv: "0bc1ffcb845dcbdcbe0c206283ada2a0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.352777 11853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:07:58.355225 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.356431 11853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:58.356561 11853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "619b66d8e74f4b1abc285b03311a2b80"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "80146bc8fe42ef3ab6ce019627864c6e"
server_key_iv: "0bc1ffcb845dcbdcbe0c206283ada2a0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.356673 11853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:58.361316 11853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:58.361930 11853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:58.362123 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:58.368911 11853 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34481
I20260808 02:07:58.368920 11936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34481 every 8 connection(s)
I20260808 02:07:58.369869 11853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:58.371907 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11853
I20260808 02:07:58.372045 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:58.372370 5957 external_mini_cluster.cc:1428] Setting key aa3e41e2d468c5109ce42bbc0dac6644
I20260808 02:07:58.373757 11939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:58.378319 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.373973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.5.209.126:34481)
0808 02:07:58.374674 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.374691 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.374728 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:07:58.375391 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.375407 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.375765 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.376124 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:58.376993 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:58.376997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:58.377012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:58.377022 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:58.377080 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:58.377096 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:58.377255 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.377332 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.377632 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260808 02:07:58.380185 11939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: Bootstrap starting.
I20260808 02:07:58.382714 11939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:58.383460 11939 log.cc:824] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:58.385627 11939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: No bootstrap required, opened a new log
I20260808 02:07:58.389494 11939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } }
I20260808 02:07:58.389776 11939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:58.389828 11939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619b66d8e74f4b1abc285b03311a2b80, State: Initialized, Role: FOLLOWER
I20260808 02:07:58.390257 11939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } }
I20260808 02:07:58.390385 11939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:58.390427 11939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:58.390494 11939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:58.391695 11939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } }
I20260808 02:07:58.392081 11939 leader_election.cc:302] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619b66d8e74f4b1abc285b03311a2b80; no voters:
I20260808 02:07:58.392436 11939 leader_election.cc:288] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:58.392589 11945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:58.392823 11945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [term 1 LEADER]: Becoming Leader. State: Replica: 619b66d8e74f4b1abc285b03311a2b80, State: Running, Role: LEADER
I20260808 02:07:58.393127 11945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } }
I20260808 02:07:58.393805 11939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:58.395066 11946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619b66d8e74f4b1abc285b03311a2b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } } }
I20260808 02:07:58.395067 11947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619b66d8e74f4b1abc285b03311a2b80. Latest consensus state: current_term: 1 leader_uuid: "619b66d8e74f4b1abc285b03311a2b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619b66d8e74f4b1abc285b03311a2b80" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34481 } } }
I20260808 02:07:58.395236 11947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:58.395236 11946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:58.395684 11957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:58.399484 11957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:58.404943 11957 catalog_manager.cc:1347] Generated new cluster ID: e0df6af0238149f096936fc20e378676
I20260808 02:07:58.405005 11957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:58.414786 11957 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:58.415076 11957 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:58.423286 11957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: Generated new TSK 0
I20260808 02:07:58.423920 11957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:07:58.428660 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--builtin_ntp_servers=127.5.209.84:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:07:58.546777 11969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:58.547030 11969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:58.547070 11969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:58.550983 11969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:58.551036 11969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:58.551091 11969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:07:58.555809 11969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:58.556916 11969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:58.558326 11969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:58.568333 11983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.568445 11969 server_base.cc:1034] running on GCE node
W20260808 02:07:58.568372 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:58.568352 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.568954 11969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:58.569660 11969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:58.570837 11969 hybrid_clock.cc:648] HybridClock initialized: now 1786154878570820 us; error 37 us; skew 500 ppm
I20260808 02:07:58.572696 11969 webserver.cc:460] Webserver started at http://127.5.209.65:36831/ using document root <none> and password file <none>
I20260808 02:07:58.573293 11969 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:58.573417 11969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:58.573674 11969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:58.575820 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "62e0e2a23396401e86a63ee5915fd784"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "f73a4c0450ec26f985cb6f48e44b998a"
server_key_iv: "d8a698cfefe24cb946c769107c0f12d0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.576416 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "62e0e2a23396401e86a63ee5915fd784"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "f73a4c0450ec26f985cb6f48e44b998a"
server_key_iv: "d8a698cfefe24cb946c769107c0f12d0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.580255 11969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 02:07:58.582896 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.584182 11969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:58.584323 11969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "62e0e2a23396401e86a63ee5915fd784"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "f73a4c0450ec26f985cb6f48e44b998a"
server_key_iv: "d8a698cfefe24cb946c769107c0f12d0"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.584442 11969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:58.589236 11969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:58.589884 11969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:58.590080 11969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:58.590653 11969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:58.591615 11969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:58.591687 11969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.591760 11969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:58.591802 11969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.601518 11969 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:41727
I20260808 02:07:58.601574 12149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:41727 every 8 connection(s)
I20260808 02:07:58.602481 11969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:07:58.606076 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 11969
I20260808 02:07:58.606228 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:07:58.606532 5957 external_mini_cluster.cc:1428] Setting key dd10662e7ac60cd3afe14562ce61b3a0
I20260808 02:07:58.608934 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--builtin_ntp_servers=127.5.209.84:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:07:58.608982 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.604330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:43379 (local address 127.5.209.126:34481)
0808 02:07:58.604482 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.604486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.605582 (+ 1096us) server_negotiation.cc:407] Connection header received
0808 02:07:58.607139 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.607142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.607181 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.607246 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:58.608545 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:58.608548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:58.608550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:58.608560 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:58.608581 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:58.608589 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:58.608664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.608847 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.608881 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:07:58.610157 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.604742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34481 (local address 127.5.209.65:43379)
0808 02:07:58.605441 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.605480 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:58.606892 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:58.607409 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:58.607435 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:58.607542 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:58.608245 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:58.608247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:58.608302 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:58.608389 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:58.608685 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:58.608698 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:58.608706 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:07:58.608892 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 02:07:58.609147 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260808 02:07:58.611578 12152 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34481
I20260808 02:07:58.611840 12152 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:58.612254 12152 heartbeater.cc:511] Master 127.5.209.126:34481 requested a full tablet report, sending...
I20260808 02:07:58.614025 11891 ts_manager.cc:194] Registered new tserver with Master: 62e0e2a23396401e86a63ee5915fd784 (127.5.209.65:41727)
W20260808 02:07:58.726310 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:58.726598 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:58.726675 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:58.731276 12156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:58.731348 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:58.731426 12156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:07:58.736352 12156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:58.737440 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:58.738806 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:58.749457 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.749603 12156 server_base.cc:1034] running on GCE node
W20260808 02:07:58.749507 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:58.749672 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.750227 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:58.750857 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:58.752058 12156 hybrid_clock.cc:648] HybridClock initialized: now 1786154878752047 us; error 31 us; skew 500 ppm
I20260808 02:07:58.753991 12156 webserver.cc:460] Webserver started at http://127.5.209.66:38141/ using document root <none> and password file <none>
I20260808 02:07:58.754580 12156 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:58.754675 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:58.754915 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:58.757026 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "72c344180afc4d47a132afbe59205bb2"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "338d8ac5c9979573840cd3f7447910a6"
server_key_iv: "2a9da544f2f0c25f242349db8df87f79"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.757643 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "72c344180afc4d47a132afbe59205bb2"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "338d8ac5c9979573840cd3f7447910a6"
server_key_iv: "2a9da544f2f0c25f242349db8df87f79"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.761678 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:07:58.764066 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.765312 12156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:07:58.765502 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "72c344180afc4d47a132afbe59205bb2"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "338d8ac5c9979573840cd3f7447910a6"
server_key_iv: "2a9da544f2f0c25f242349db8df87f79"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.765610 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:58.770203 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:58.770797 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:58.770994 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:58.771549 12156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:58.772496 12156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:58.772545 12156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.772612 12156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:58.772650 12156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.781857 12156 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:41403
I20260808 02:07:58.781883 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:41403 every 8 connection(s)
I20260808 02:07:58.782763 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:07:58.786490 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 12156
I20260808 02:07:58.786630 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:07:58.786916 5957 external_mini_cluster.cc:1428] Setting key 19a7a0efe3bdbf59ae26f9dd6e533a8c
I20260808 02:07:58.789098 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--builtin_ntp_servers=127.5.209.84:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:07:58.789225 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.784608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:36359 (local address 127.5.209.126:34481)
0808 02:07:58.784751 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.784755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.785833 (+ 1078us) server_negotiation.cc:407] Connection header received
0808 02:07:58.787302 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.787305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.787353 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.787478 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:58.788745 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:58.788749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:58.788752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:58.788763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:58.788785 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:58.788801 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:58.788893 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.789047 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.789091 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:07:58.790086 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.784994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34481 (local address 127.5.209.66:36359)
0808 02:07:58.785692 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.785731 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:58.787093 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:58.787716 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:58.787725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:58.787827 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:58.788461 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:58.788462 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:58.788500 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:58.788598 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:58.788881 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:58.788891 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:58.788897 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:07:58.789077 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 02:07:58.789319 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260808 02:07:58.791970 12322 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34481
I20260808 02:07:58.792217 12322 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:58.792732 12322 heartbeater.cc:511] Master 127.5.209.126:34481 requested a full tablet report, sending...
I20260808 02:07:58.793926 11891 ts_manager.cc:194] Registered new tserver with Master: 72c344180afc4d47a132afbe59205bb2 (127.5.209.66:41403)
W20260808 02:07:58.911423 12326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:58.911700 12326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:58.911742 12326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:58.915725 12326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:58.915781 12326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:58.915839 12326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:07:58.920612 12326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:58.921778 12326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:58.923345 12326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:58.933432 12332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.933488 12326 server_base.cc:1034] running on GCE node
W20260808 02:07:58.933439 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:58.933439 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:58.934105 12326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:58.934760 12326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:58.935923 12326 hybrid_clock.cc:648] HybridClock initialized: now 1786154878935847 us; error 88 us; skew 500 ppm
I20260808 02:07:58.937980 12326 webserver.cc:460] Webserver started at http://127.5.209.67:45341/ using document root <none> and password file <none>
I20260808 02:07:58.938575 12326 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:58.938640 12326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:58.938902 12326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:58.940932 12326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "852d62bb3a864087ad1ff2f7e7dd03aa"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "8e2c4c43f4c9b384202d7615c7b43c5a"
server_key_iv: "60416717582d5e2a68999928da7dbc65"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.941578 12326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "852d62bb3a864087ad1ff2f7e7dd03aa"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "8e2c4c43f4c9b384202d7615c7b43c5a"
server_key_iv: "60416717582d5e2a68999928da7dbc65"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.945297 12326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:07:58.947902 12344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.949105 12326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:07:58.949236 12326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "852d62bb3a864087ad1ff2f7e7dd03aa"
format_stamp: "Formatted at 2026-08-08 02:07:58 on dist-test-slave-bw88"
server_key: "8e2c4c43f4c9b384202d7615c7b43c5a"
server_key_iv: "60416717582d5e2a68999928da7dbc65"
server_key_version: "encryptionkey@0"
I20260808 02:07:58.949371 12326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:58.953907 12326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:58.954489 12326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:58.954669 12326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:58.955221 12326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:07:58.956130 12326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:07:58.956199 12326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.956267 12326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:07:58.956306 12326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:58.965509 12326 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:35349
I20260808 02:07:58.965572 12505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:35349 every 8 connection(s)
I20260808 02:07:58.966470 12326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:07:58.966814 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 12326
I20260808 02:07:58.966972 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:07:58.967350 5957 external_mini_cluster.cc:1428] Setting key a4066669dee399ae0a075c3fed9e1670
I20260808 02:07:58.973397 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.968543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59745 (local address 127.5.209.126:34481)
0808 02:07:58.969520 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.969525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.970166 (+ 641us) server_negotiation.cc:407] Connection header received
0808 02:07:58.971559 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.971562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.971606 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.971685 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:58.972852 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:58.972855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:58.972858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:58.972870 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:58.972892 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:58.972902 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:58.972995 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.973180 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.973228 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862}
I20260808 02:07:58.974238 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.969052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34481 (local address 127.5.209.67:59745)
0808 02:07:58.970005 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:58.970062 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:58.971335 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:58.971857 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:58.971864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:58.971953 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:58.972594 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:58.972595 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:58.972636 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:58.972706 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:58.972996 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:58.973007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:58.973014 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:07:58.973210 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 02:07:58.973488 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":181,"threads_started":1}
I20260808 02:07:58.975274 12507 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34481
I20260808 02:07:58.975481 12507 heartbeater.cc:464] Registering TS with master...
I20260808 02:07:58.975845 12507 heartbeater.cc:511] Master 127.5.209.126:34481 requested a full tablet report, sending...
I20260808 02:07:58.976765 11891 ts_manager.cc:194] Registered new tserver with Master: 852d62bb3a864087ad1ff2f7e7dd03aa (127.5.209.67:35349)
I20260808 02:07:58.981798 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:07:58.985165 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.983693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.5.209.126:34481)
0808 02:07:58.983831 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:58.983835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:58.984016 (+ 181us) server_negotiation.cc:407] Connection header received
0808 02:07:58.984100 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:58.984102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:58.984134 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:58.984191 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:58.984841 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:58.984843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:58.984845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:58.984853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:58.984868 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:58.984874 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:58.984936 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:58.984975 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:07:58.985014 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:07:58.988212 11891 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:07:58.991400 11891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:07:59.003400 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.000610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41403 (local address 127.0.0.1:33082)
0808 02:07:59.000875 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.000888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.000958 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.002225 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.002228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.002243 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:59.002672 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:59.002673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:59.002682 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:59.002704 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:59.003031 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:59.003040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:59.003046 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:07:59.003167 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:07:59.003240 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
I20260808 02:07:59.003468 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.001047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.5.209.66:41403)
0808 02:07:59.001790 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.001803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.001828 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:07:59.001895 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.001900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.002070 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.002202 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.002834 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.002840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.002853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.002871 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.002918 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.002951 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.003065 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.003198 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.003277 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":155,"threads_started":1}
I20260808 02:07:59.003734 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.000645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.5.209.67:35349)
0808 02:07:59.001291 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.001303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.001328 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:07:59.001408 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.001415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.001624 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.001765 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.003021 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.003028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.003040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.003053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.003101 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.003144 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.003241 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.003466 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.003542 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":96,"threads_started":1}
I20260808 02:07:59.003856 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.999907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:35349 (local address 127.0.0.1:51920)
0808 02:07:59.000409 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.000422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.000479 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.002388 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.002392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.002412 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:59.002844 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:59.002846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:59.002859 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:59.002886 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:59.003283 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:59.003286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:59.003290 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:07:59.003356 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:07:59.003385 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":104,"threads_started":1}
I20260808 02:07:59.004280 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.999834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41727 (local address 127.0.0.1:52568)
0808 02:07:59.000302 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.000356 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.000465 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.001251 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.001258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.001357 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:59.001941 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:59.001943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:59.001963 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:59.002031 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:59.004045 (+ 2014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:59.004048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:59.004052 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:07:59.004130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:07:59.004164 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":84,"threads_started":1}
I20260808 02:07:59.004845 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:58.999975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.5.209.65:41727)
0808 02:07:59.000709 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.000718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.000760 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:07:59.000822 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.000827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.001029 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.002200 (+ 1171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.003281 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.003286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.003295 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.003304 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.003344 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.003369 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.004571 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.004620 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.004685 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":154,"threads_started":1}
I20260808 02:07:59.007135 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 (DEFAULT_TABLE table=test-table [id=babf15bf592243f5ab80fcfbeea7bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:59.007136 12060 tablet_service.cc:1467] Processing CreateTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 (DEFAULT_TABLE table=test-table [id=babf15bf592243f5ab80fcfbeea7bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:59.008159 12060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b4b37f6c4c4acab4933bd4ba70c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:59.008180 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b4b37f6c4c4acab4933bd4ba70c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:59.009011 12231 tablet_service.cc:1467] Processing CreateTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 (DEFAULT_TABLE table=test-table [id=babf15bf592243f5ab80fcfbeea7bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:07:59.010072 12231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b4b37f6c4c4acab4933bd4ba70c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:59.014575 12530 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Bootstrap starting.
I20260808 02:07:59.016414 12531 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Bootstrap starting.
I20260808 02:07:59.016789 12530 tablet_bootstrap.cc:654] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:59.017176 12532 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Bootstrap starting.
I20260808 02:07:59.017910 12530 log.cc:824] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:59.018671 12531 tablet_bootstrap.cc:654] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:59.019429 12532 tablet_bootstrap.cc:654] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:59.019449 12531 log.cc:824] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:59.020287 12532 log.cc:824] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:59.020404 12530 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: No bootstrap required, opened a new log
I20260808 02:07:59.020604 12530 ts_tablet_manager.cc:1397] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:07:59.021680 12531 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: No bootstrap required, opened a new log
I20260808 02:07:59.021860 12531 ts_tablet_manager.cc:1397] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:07:59.022662 12532 tablet_bootstrap.cc:492] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: No bootstrap required, opened a new log
I20260808 02:07:59.022847 12532 ts_tablet_manager.cc:1397] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:07:59.024739 12530 raft_consensus.cc:348] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.024969 12530 raft_consensus.cc:374] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:59.025027 12530 raft_consensus.cc:729] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e0e2a23396401e86a63ee5915fd784, State: Initialized, Role: FOLLOWER
I20260808 02:07:59.025549 12530 consensus_queue.cc:260] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.026243 12152 heartbeater.cc:503] Master 127.5.209.126:34481 was elected leader, sending a full tablet report...
I20260808 02:07:59.026546 12530 ts_tablet_manager.cc:1428] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 02:07:59.026394 12531 raft_consensus.cc:348] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.026630 12531 raft_consensus.cc:374] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:59.026693 12531 raft_consensus.cc:729] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852d62bb3a864087ad1ff2f7e7dd03aa, State: Initialized, Role: FOLLOWER
I20260808 02:07:59.027108 12531 consensus_queue.cc:260] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.027730 12507 heartbeater.cc:503] Master 127.5.209.126:34481 was elected leader, sending a full tablet report...
I20260808 02:07:59.028071 12531 ts_tablet_manager.cc:1428] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:07:59.028008 12532 raft_consensus.cc:348] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.028242 12532 raft_consensus.cc:374] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:59.028293 12532 raft_consensus.cc:729] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c344180afc4d47a132afbe59205bb2, State: Initialized, Role: FOLLOWER
I20260808 02:07:59.028726 12532 consensus_queue.cc:260] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.029511 12322 heartbeater.cc:503] Master 127.5.209.126:34481 was elected leader, sending a full tablet report...
I20260808 02:07:59.029784 12532 ts_tablet_manager.cc:1428] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260808 02:07:59.033697 12323 tablet.cc:2369] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:07:59.077625 12539 raft_consensus.cc:482] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:07:59.077950 12539 raft_consensus.cc:504] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.079254 12539 leader_election.cc:288] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62e0e2a23396401e86a63ee5915fd784 (127.5.209.65:41727), 72c344180afc4d47a132afbe59205bb2 (127.5.209.66:41403)
I20260808 02:07:59.081305 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.079675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:36071 (local address 127.5.209.66:41403)
0808 02:07:59.079826 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.079833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.080094 (+ 261us) server_negotiation.cc:407] Connection header received
0808 02:07:59.080168 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.080170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.080203 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.080266 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.080854 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.080858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.080861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.080874 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.080896 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.080907 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.081001 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.081109 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.081169 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:07:59.081305 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.079616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:45691 (local address 127.5.209.65:41727)
0808 02:07:59.079742 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.079747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.079761 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:07:59.079905 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.079909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.079956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.080026 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.080854 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.080858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.080861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.080874 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.080896 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.080907 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.081001 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.081118 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.081169 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:07:59.081306 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.079457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41727 (local address 127.5.209.67:45691)
0808 02:07:59.079665 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.079681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.079763 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.080085 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.080088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.080103 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:59.080643 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:59.080646 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:59.080661 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:59.080697 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:59.081020 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:59.081022 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:59.081025 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:07:59.081101 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:07:59.081165 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260808 02:07:59.081368 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.079515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41403 (local address 127.5.209.67:36071)
0808 02:07:59.079966 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.079978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.080088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.080320 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.080329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.080341 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:07:59.080702 (+ 361us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:07:59.080704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:07:59.080712 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:07:59.080735 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:07:59.081011 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:07:59.081015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:07:59.081018 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 02:07:59.081101 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:07:59.081178 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":31,"thread_start_us":172,"threads_started":1}
I20260808 02:07:59.082297 12260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b4b37f6c4c4acab4933bd4ba70c011" candidate_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c344180afc4d47a132afbe59205bb2" is_pre_election: true
I20260808 02:07:59.082337 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b4b37f6c4c4acab4933bd4ba70c011" candidate_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e0e2a23396401e86a63ee5915fd784" is_pre_election: true
I20260808 02:07:59.082628 12260 raft_consensus.cc:2454] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852d62bb3a864087ad1ff2f7e7dd03aa in term 0.
I20260808 02:07:59.082618 12087 raft_consensus.cc:2454] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852d62bb3a864087ad1ff2f7e7dd03aa in term 0.
I20260808 02:07:59.083127 12346 leader_election.cc:302] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c344180afc4d47a132afbe59205bb2, 852d62bb3a864087ad1ff2f7e7dd03aa; no voters:
I20260808 02:07:59.083436 12539 raft_consensus.cc:2781] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:07:59.083555 12539 raft_consensus.cc:482] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:07:59.083617 12539 raft_consensus.cc:3037] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:59.084705 12539 raft_consensus.cc:504] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.085177 12539 leader_election.cc:288] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [CANDIDATE]: Term 1 election: Requested vote from peers 62e0e2a23396401e86a63ee5915fd784 (127.5.209.65:41727), 72c344180afc4d47a132afbe59205bb2 (127.5.209.66:41403)
I20260808 02:07:59.085561 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b4b37f6c4c4acab4933bd4ba70c011" candidate_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e0e2a23396401e86a63ee5915fd784"
I20260808 02:07:59.085685 12260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b4b37f6c4c4acab4933bd4ba70c011" candidate_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c344180afc4d47a132afbe59205bb2"
I20260808 02:07:59.085697 12087 raft_consensus.cc:3037] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:59.085835 12260 raft_consensus.cc:3037] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:59.087324 12087 raft_consensus.cc:2454] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852d62bb3a864087ad1ff2f7e7dd03aa in term 1.
I20260808 02:07:59.087427 12260 raft_consensus.cc:2454] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852d62bb3a864087ad1ff2f7e7dd03aa in term 1.
I20260808 02:07:59.087738 12348 leader_election.cc:302] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62e0e2a23396401e86a63ee5915fd784, 852d62bb3a864087ad1ff2f7e7dd03aa; no voters:
I20260808 02:07:59.087990 12539 raft_consensus.cc:2781] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:59.088284 12539 raft_consensus.cc:686] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 1 LEADER]: Becoming Leader. State: Replica: 852d62bb3a864087ad1ff2f7e7dd03aa, State: Running, Role: LEADER
I20260808 02:07:59.088728 12539 consensus_queue.cc:237] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } }
I20260808 02:07:59.092453 11890 catalog_manager.cc:5495] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 852d62bb3a864087ad1ff2f7e7dd03aa (127.5.209.67). New cstate: current_term: 1 leader_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852d62bb3a864087ad1ff2f7e7dd03aa" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260808 02:07:59.103456 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.101929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.5.209.67:35349)
0808 02:07:59.102084 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.102087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.102100 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:07:59.102266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.102268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.102326 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.102395 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.103048 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.103051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.103053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.103064 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.103082 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.103090 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.103162 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.103240 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.103274 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 02:07:59.103816 12154 tablet.cc:2369] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:07:59.109438 12087 raft_consensus.cc:1261] T 60b4b37f6c4c4acab4933bd4ba70c011 P 62e0e2a23396401e86a63ee5915fd784 [term 1 FOLLOWER]: Refusing update from remote peer 852d62bb3a864087ad1ff2f7e7dd03aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:59.109448 12260 raft_consensus.cc:1261] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 1 FOLLOWER]: Refusing update from remote peer 852d62bb3a864087ad1ff2f7e7dd03aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:07:59.110159 12545 consensus_queue.cc:1035] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62e0e2a23396401e86a63ee5915fd784" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:59.110337 12539 consensus_queue.cc:1035] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c344180afc4d47a132afbe59205bb2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:07:59.118564 12551 mvcc.cc:205] Tried to move back new op lower bound from 7316090384824430592 to 7316090384750481408. Current Snapshot: MvccSnapshot[applied={T|T < 7316090384824430592}]
I20260808 02:07:59.118985 12553 mvcc.cc:205] Tried to move back new op lower bound from 7316090384824430592 to 7316090384750481408. Current Snapshot: MvccSnapshot[applied={T|T < 7316090384824430592}]
I20260808 02:07:59.126456 11889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:59.126853 11889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:07:59.129634 11889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 619b66d8e74f4b1abc285b03311a2b80: Sending DeleteTablet for 3 replicas of tablet 60b4b37f6c4c4acab4933bd4ba70c011
I20260808 02:07:59.130434 12060 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:59 UTC) from {username='slave'} at 127.0.0.1:52568
I20260808 02:07:59.130504 12231 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:59 UTC) from {username='slave'} at 127.0.0.1:33082
I20260808 02:07:59.130506 12409 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b4b37f6c4c4acab4933bd4ba70c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:07:59 UTC) from {username='slave'} at 127.0.0.1:51920
I20260808 02:07:59.130972 12564 tablet_replica.cc:330] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: stopping tablet replica
I20260808 02:07:59.131137 12563 tablet_replica.cc:330] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: stopping tablet replica
I20260808 02:07:59.131289 12564 raft_consensus.cc:2229] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:07:59.131524 12563 raft_consensus.cc:2229] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:07:59.131713 12564 raft_consensus.cc:2258] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:59.131760 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11969
I20260808 02:07:59.139827 12564 ts_tablet_manager.cc:1905] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:59.139981 12563 raft_consensus.cc:2258] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:07:59.141641 12563 ts_tablet_manager.cc:1905] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:07:59.143640 12564 ts_tablet_manager.cc:1918] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:59.143766 12564 log.cc:1196] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/60b4b37f6c4c4acab4933bd4ba70c011
I20260808 02:07:59.144169 12564 ts_tablet_manager.cc:1939] T 60b4b37f6c4c4acab4933bd4ba70c011 P 852d62bb3a864087ad1ff2f7e7dd03aa: Deleting consensus metadata
I20260808 02:07:59.144757 12563 ts_tablet_manager.cc:1918] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:07:59.144919 12563 log.cc:1196] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/60b4b37f6c4c4acab4933bd4ba70c011
I20260808 02:07:59.145311 12563 ts_tablet_manager.cc:1939] T 60b4b37f6c4c4acab4933bd4ba70c011 P 72c344180afc4d47a132afbe59205bb2: Deleting consensus metadata
I20260808 02:07:59.145610 11872 catalog_manager.cc:4841] TS 852d62bb3a864087ad1ff2f7e7dd03aa (127.5.209.67:35349): tablet 60b4b37f6c4c4acab4933bd4ba70c011 (table test-table [id=babf15bf592243f5ab80fcfbeea7bf36]) successfully deleted
I20260808 02:07:59.146660 11873 catalog_manager.cc:4841] TS 72c344180afc4d47a132afbe59205bb2 (127.5.209.66:41403): tablet 60b4b37f6c4c4acab4933bd4ba70c011 (table test-table [id=babf15bf592243f5ab80fcfbeea7bf36]) successfully deleted
W20260808 02:07:59.148725 11874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:41727 (error 108)
I20260808 02:07:59.149595 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 12156
I20260808 02:07:59.149886 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.149565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41727 (local address 127.0.0.1:52576)
0808 02:07:59.149708 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.149790 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 02:07:59.154042 11874 catalog_manager.cc:4584] TS 62e0e2a23396401e86a63ee5915fd784 (127.5.209.65:41727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60b4b37f6c4c4acab4933bd4ba70c011: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41727: connect: Connection refused (error 111)
I20260808 02:07:59.162076 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 12326
I20260808 02:07:59.172995 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 11853
2026-08-08T02:07:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1000 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 02:07:59.198133 5957 mini_oidc.cc:152] Starting JWKS server
I20260808 02:07:59.199890 5957 webserver.cc:460] Webserver started at https://127.0.0.1:36895/ using document root <none> and password file <none>
I20260808 02:07:59.200093 5957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:07:59.200377 5957 webserver.cc:460] Webserver started at http://127.0.0.1:38915/ using document root <none> and password file <none>
2026-08-08T02:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:59Z Disabled control of system clock
I20260808 02:07:59.210899 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:44175
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:44175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/testchainca.cert with env {}
W20260808 02:07:59.328214 12575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:59.328466 12575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:59.328511 12575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:59.332367 12575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:59.332422 12575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:59.332438 12575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:59.332454 12575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:59.332470 12575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:07:59.332490 12575 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:07:59.332504 12575 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36895/jwks.json
I20260808 02:07:59.337764 12575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:44175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:44175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:59.338743 12575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:59.339704 12575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:07:59.348287 12575 server_base.cc:1034] running on GCE node
W20260808 02:07:59.348223 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:59.348258 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:59.348234 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:59.349023 12575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:59.350178 12575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:59.351370 12575 hybrid_clock.cc:648] HybridClock initialized: now 1786154879351351 us; error 31 us; skew 500 ppm
I20260808 02:07:59.353195 12575 webserver.cc:460] Webserver started at http://127.5.209.126:36473/ using document root <none> and password file <none>
I20260808 02:07:59.353865 12575 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:59.353960 12575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:59.354203 12575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:59.356225 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "6508800e457148eca4f344e663ee4588"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "3f02e26c974acf320379c1ac0c56c7bf"
server_key_iv: "18cd0151d4b1dce0099ca434662890ba"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.356799 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "6508800e457148eca4f344e663ee4588"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "3f02e26c974acf320379c1ac0c56c7bf"
server_key_iv: "18cd0151d4b1dce0099ca434662890ba"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.360378 12575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:07:59.362782 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:59.363920 12575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:07:59.364061 12575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "6508800e457148eca4f344e663ee4588"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "3f02e26c974acf320379c1ac0c56c7bf"
server_key_iv: "18cd0151d4b1dce0099ca434662890ba"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.364177 12575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:59.383531 12575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:59.384275 12575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:59.384486 12575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:59.392135 12575 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:44175
I20260808 02:07:59.392195 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:44175 every 8 connection(s)
I20260808 02:07:59.393163 12575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:59.396602 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:59.397164 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 12575
I20260808 02:07:59.397318 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:59.397635 5957 external_mini_cluster.cc:1428] Setting key 1528c846bd60e5182953eb86267ced95
I20260808 02:07:59.403559 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588: Bootstrap starting.
I20260808 02:07:59.406131 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:59.407016 12674 log.cc:824] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:59.408156 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.399034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.5.209.126:44175)
0808 02:07:59.399710 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.399729 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.399777 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:07:59.400645 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.400700 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.401222 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.401631 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.402063 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.403564 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.404484 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.405174 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.405354 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.406185 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.406192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.406209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.406732 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.406800 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.406827 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.406938 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.407061 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.407328 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260808 02:07:59.409400 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588: No bootstrap required, opened a new log
I20260808 02:07:59.411010 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:07:59.413445 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } }
I20260808 02:07:59.413681 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:59.413759 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6508800e457148eca4f344e663ee4588, State: Initialized, Role: FOLLOWER
I20260808 02:07:59.414180 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } }
I20260808 02:07:59.414333 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:59.414422 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:59.414553 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:59.415572 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } }
I20260808 02:07:59.415910 12674 leader_election.cc:302] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6508800e457148eca4f344e663ee4588; no voters:
I20260808 02:07:59.416230 12674 leader_election.cc:288] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:59.416380 12680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:59.416610 12680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [term 1 LEADER]: Becoming Leader. State: Replica: 6508800e457148eca4f344e663ee4588, State: Running, Role: LEADER
I20260808 02:07:59.416915 12680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } }
I20260808 02:07:59.417479 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:59.418721 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6508800e457148eca4f344e663ee4588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } } }
I20260808 02:07:59.418857 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:59.419209 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6508800e457148eca4f344e663ee4588. Latest consensus state: current_term: 1 leader_uuid: "6508800e457148eca4f344e663ee4588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508800e457148eca4f344e663ee4588" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44175 } } }
I20260808 02:07:59.419332 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588 [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:59.419276 12694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:59.422333 12694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:59.427798 12694 catalog_manager.cc:1347] Generated new cluster ID: 8f2db2db8f6541b9a7632e167719a3c8
I20260808 02:07:59.427881 12694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:59.437791 12694 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:59.438697 12694 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:59.451208 12694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6508800e457148eca4f344e663ee4588: Generated new TSK 0
I20260808 02:07:59.451987 12694 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:07:59.487870 12715 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 02:07:59.488070 12715 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:07:59.488597 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.471013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.5.209.126:44175)
0808 02:07:59.471165 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.471170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.471402 (+ 232us) server_negotiation.cc:407] Connection header received
0808 02:07:59.471641 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.471644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.471691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.471776 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:07:59.472079 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.472626 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.473877 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.474116 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.474194 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.474605 (+ 411us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:07:59.488239 (+ 13634us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:07:59.488481 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 02:07:59.488700 12677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 02:07:59.490262 12712 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:07:59.470943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:44175 (local address 127.0.0.1:33048)
0808 02:07:59.471307 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.471324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.471484 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.471821 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.471825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.471833 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:07:59.471971 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:59.471977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.472747 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.472751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:59.473765 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:59.473776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.474235 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.474239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:59.474337 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.474389 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:07:59.488493 (+ 14104us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:07:59.488755 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.209.126:44175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.209.126:44175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 02:07:59.491128 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 12575
2026-08-08T02:07:59Z chronyd exiting
I20260808 02:07:59.536134 5957 test_util.cc:175] -----------------------------------------------
I20260808 02:07:59.536233 5957 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786154850122309-5957-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 02:07:59.537563 5957 mini_oidc.cc:152] Starting JWKS server
I20260808 02:07:59.538518 5957 webserver.cc:460] Webserver started at https://127.0.0.1:40861/ using document root <none> and password file <none>
I20260808 02:07:59.538666 5957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:07:59.538969 5957 webserver.cc:460] Webserver started at http://127.0.0.1:33587/ using document root <none> and password file <none>
2026-08-08T02:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:59Z Disabled control of system clock
I20260808 02:07:59.549759 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42031
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:42031
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/testchainca.cert with env {}
W20260808 02:07:59.667785 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:07:59.668113 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:07:59.668196 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:07:59.672180 12722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:07:59.672290 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:07:59.672325 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:07:59.672369 12722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:07:59.672407 12722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:07:59.672447 12722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:07:59.672482 12722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40861/jwks.json
I20260808 02:07:59.677829 12722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:42031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:07:59.678992 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:07:59.680068 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:07:59.689213 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:59.689399 12722 server_base.cc:1034] running on GCE node
W20260808 02:07:59.689173 12735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:07:59.689173 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:07:59.690131 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:07:59.691286 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:07:59.692497 12722 hybrid_clock.cc:648] HybridClock initialized: now 1786154879692464 us; error 44 us; skew 500 ppm
I20260808 02:07:59.694725 12722 webserver.cc:460] Webserver started at http://127.5.209.126:33521/ using document root <none> and password file <none>
I20260808 02:07:59.695367 12722 fs_manager.cc:359] Metadata directory not provided
I20260808 02:07:59.695463 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:07:59.695684 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:07:59.697831 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "916e31d3be444aefa4df1809cac1c32b"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "e787b4191e20964b57f9711dfbb99ad8"
server_key_iv: "b6f99bbe66e80768131535de7ab6042e"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.698679 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "916e31d3be444aefa4df1809cac1c32b"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "e787b4191e20964b57f9711dfbb99ad8"
server_key_iv: "b6f99bbe66e80768131535de7ab6042e"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.702695 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:07:59.705374 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:07:59.706851 12722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:07:59.707027 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "916e31d3be444aefa4df1809cac1c32b"
format_stamp: "Formatted at 2026-08-08 02:07:59 on dist-test-slave-bw88"
server_key: "e787b4191e20964b57f9711dfbb99ad8"
server_key_iv: "b6f99bbe66e80768131535de7ab6042e"
server_key_version: "encryptionkey@0"
I20260808 02:07:59.707157 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:07:59.732123 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:07:59.732798 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:07:59.732999 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:07:59.740854 12722 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:42031
I20260808 02:07:59.740865 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:42031 every 8 connection(s)
I20260808 02:07:59.741891 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:07:59.745088 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:07:59.745914 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 12722
I20260808 02:07:59.746059 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:07:59.746392 5957 external_mini_cluster.cc:1428] Setting key cdad9e33340abc617dd35b37d193b0f2
I20260808 02:07:59.752069 12824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b: Bootstrap starting.
I20260808 02:07:59.754940 12824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b: Neither blocks nor log segments found. Creating new log.
I20260808 02:07:59.755841 12824 log.cc:824] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:07:59.756657 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.748308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.5.209.126:42031)
0808 02:07:59.749232 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.749254 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.749301 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:07:59.750002 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.750018 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.750415 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.750700 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:07:59.751040 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.751967 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.753174 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.753669 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.753788 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.754654 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:07:59.754659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:07:59.754671 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:07:59.755109 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:07:59.755170 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:07:59.755192 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:07:59.755270 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:07:59.755435 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:07:59.755736 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":140,"threads_started":1}
I20260808 02:07:59.758229 12824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b: No bootstrap required, opened a new log
I20260808 02:07:59.759604 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:07:59.761651 12824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } }
I20260808 02:07:59.761919 12824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:07:59.762024 12824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916e31d3be444aefa4df1809cac1c32b, State: Initialized, Role: FOLLOWER
I20260808 02:07:59.762395 12824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } }
I20260808 02:07:59.762528 12824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:07:59.762573 12824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:07:59.762643 12824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:07:59.763634 12824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } }
I20260808 02:07:59.763970 12824 leader_election.cc:302] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916e31d3be444aefa4df1809cac1c32b; no voters:
I20260808 02:07:59.764241 12824 leader_election.cc:288] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:07:59.764374 12830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:07:59.764626 12830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [term 1 LEADER]: Becoming Leader. State: Replica: 916e31d3be444aefa4df1809cac1c32b, State: Running, Role: LEADER
I20260808 02:07:59.764943 12830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } }
I20260808 02:07:59.765532 12824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:07:59.766736 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916e31d3be444aefa4df1809cac1c32b. Latest consensus state: current_term: 1 leader_uuid: "916e31d3be444aefa4df1809cac1c32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } } }
I20260808 02:07:59.766893 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:59.767189 12831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916e31d3be444aefa4df1809cac1c32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916e31d3be444aefa4df1809cac1c32b" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42031 } } }
I20260808 02:07:59.767306 12831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b [sys.catalog]: This master's current role is: LEADER
I20260808 02:07:59.767377 12841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:07:59.770582 12841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:07:59.775964 12841 catalog_manager.cc:1347] Generated new cluster ID: 51c5770ec0c44ad0b5956668ac49b1ed
I20260808 02:07:59.776032 12841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:07:59.809448 12841 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:07:59.810741 12841 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:07:59.823062 12841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916e31d3be444aefa4df1809cac1c32b: Generated new TSK 0
I20260808 02:07:59.823853 12841 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:07:59.849015 12866 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 02:07:59.849112 12866 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:07:59.849992 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:07:59.834132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.5.209.126:42031)
0808 02:07:59.834276 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:07:59.834280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:07:59.834542 (+ 262us) server_negotiation.cc:407] Connection header received
0808 02:07:59.834749 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:07:59.834752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:07:59.834800 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:07:59.834915 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:07:59.835169 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.835708 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.836634 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.836860 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.836969 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.837290 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:07:59.849466 (+ 12176us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:07:59.849809 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 02:07:59.850064 12827 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 02:07:59.850080 12865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:07:59.834071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42031 (local address 127.0.0.1:33560)
0808 02:07:59.834439 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 02:07:59.834451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:07:59.834600 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:07:59.834914 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:07:59.834918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:07:59.834926 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:07:59.835053 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:59.835059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.835837 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.835842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:59.836493 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:07:59.836501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:07:59.836985 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:07:59.836989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:07:59.837081 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:07:59.837104 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:07:59.849706 (+ 12602us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:07:59.849901 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.209.126:42031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
I20260808 02:07:59.850683 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 12722
2026-08-08T02:07:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (360 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 02:07:59.897825 5957 mini_oidc.cc:152] Starting JWKS server
I20260808 02:07:59.898720 5957 webserver.cc:460] Webserver started at https://127.0.0.1:35743/ using document root <none> and password file <none>
I20260808 02:07:59.898836 5957 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:07:59.899159 5957 webserver.cc:460] Webserver started at http://127.0.0.1:37063/ using document root <none> and password file <none>
2026-08-08T02:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:07:59Z Disabled control of system clock
I20260808 02:07:59.909555 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:32965
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:32965
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:08:00.023738 12876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:00.024158 12876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:00.024227 12876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:00.028221 12876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:00.028311 12876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:00.028340 12876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:00.028383 12876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:00.028417 12876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:08:00.028456 12876 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:08:00.028491 12876 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35743/jwks.json
I20260808 02:08:00.033232 12876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:32965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:32965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:00.034344 12876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:00.035764 12876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:00.044442 12886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:00.044447 12885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:00.044418 12888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:00.044598 12876 server_base.cc:1034] running on GCE node
I20260808 02:08:00.045153 12876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:00.046279 12876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:00.047500 12876 hybrid_clock.cc:648] HybridClock initialized: now 1786154880047469 us; error 40 us; skew 500 ppm
I20260808 02:08:00.049687 12876 webserver.cc:460] Webserver started at http://127.5.209.126:46405/ using document root <none> and password file <none>
I20260808 02:08:00.050313 12876 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:00.050419 12876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:00.050684 12876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:00.052927 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "31a712c1da6a479399450d7d8084a2d3"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "261e36e89afbfcf21f1c9a1c979ed5dc"
server_key_iv: "e1b30623e8d76a6a919c840206cbd745"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.053642 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "31a712c1da6a479399450d7d8084a2d3"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "261e36e89afbfcf21f1c9a1c979ed5dc"
server_key_iv: "e1b30623e8d76a6a919c840206cbd745"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.057582 12876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:08:00.060106 12898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:00.061394 12876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:00.061558 12876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "31a712c1da6a479399450d7d8084a2d3"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "261e36e89afbfcf21f1c9a1c979ed5dc"
server_key_iv: "e1b30623e8d76a6a919c840206cbd745"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.061673 12876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:00.088629 12876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:00.089403 12876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:00.089629 12876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:00.097378 12876 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:32965
I20260808 02:08:00.097400 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:32965 every 8 connection(s)
I20260808 02:08:00.098421 12876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:00.101817 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:00.105556 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 12876
I20260808 02:08:00.105683 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:00.105973 5957 external_mini_cluster.cc:1428] Setting key 0c341cc2b0d1d6d83536b036bdb4fff6
I20260808 02:08:00.107805 12975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3: Bootstrap starting.
I20260808 02:08:00.110081 12975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:00.110810 12975 log.cc:824] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:00.113034 12975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3: No bootstrap required, opened a new log
I20260808 02:08:00.116210 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:00.107323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.5.209.126:32965)
0808 02:08:00.108034 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:00.108050 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:00.108085 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:08:00.108731 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:00.108748 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:00.109147 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:00.109520 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:00.109965 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.111034 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.112001 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.112478 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.112669 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.113746 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:00.113753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:00.113766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:00.114206 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:00.114264 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:00.114286 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:00.114415 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:00.114630 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 02:08:00.114898 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":145,"threads_started":1}
I20260808 02:08:00.116142 12975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } }
I20260808 02:08:00.116358 12975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:00.116420 12975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a712c1da6a479399450d7d8084a2d3, State: Initialized, Role: FOLLOWER
I20260808 02:08:00.116818 12975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } }
I20260808 02:08:00.116926 12975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:00.116964 12975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:00.117054 12975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:00.117915 12975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } }
I20260808 02:08:00.118270 12975 leader_election.cc:302] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a712c1da6a479399450d7d8084a2d3; no voters:
I20260808 02:08:00.118634 12975 leader_election.cc:288] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:00.118688 12984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:00.118870 12984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 31a712c1da6a479399450d7d8084a2d3, State: Running, Role: LEADER
I20260808 02:08:00.119166 12984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } }
I20260808 02:08:00.119421 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:08:00.119850 12975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:00.120278 12986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31a712c1da6a479399450d7d8084a2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } } }
I20260808 02:08:00.120429 12986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:00.120743 12985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a712c1da6a479399450d7d8084a2d3. Latest consensus state: current_term: 1 leader_uuid: "31a712c1da6a479399450d7d8084a2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a712c1da6a479399450d7d8084a2d3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32965 } } }
I20260808 02:08:00.120860 12985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:00.121196 12988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:00.124353 12988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:00.130044 12988 catalog_manager.cc:1347] Generated new cluster ID: 65bf52e4231f49ef8d69b6153216beca
I20260808 02:08:00.130127 12988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:00.142138 12988 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:00.143062 12988 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:00.148525 12988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31a712c1da6a479399450d7d8084a2d3: Generated new TSK 0
I20260808 02:08:00.149261 12988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:08:00.163085 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:00.157671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.5.209.126:32965)
0808 02:08:00.157802 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:00.157807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:00.157993 (+ 186us) server_negotiation.cc:407] Connection header received
0808 02:08:00.158260 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:00.158263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:00.158314 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:00.158413 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:08:00.158725 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.159230 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.160526 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.160743 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.160818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.161201 (+ 383us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:08:00.162705 (+ 1504us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:08:00.162934 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 02:08:00.163159 12980 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 02:08:00.164950 13016 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:00.157530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:32965 (local address 127.0.0.1:36382)
0808 02:08:00.157880 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:00.157898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:00.158045 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:00.158456 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:00.158460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:00.158470 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:08:00.158602 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:00.158609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.159372 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.159376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:00.160350 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:00.160363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.160881 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.160886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:00.160983 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.161015 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:08:00.162892 (+ 1877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:08:00.164739 (+ 1847us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.209.126:32965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1613,"thread_start_us":114,"threads_started":1}
I20260808 02:08:00.165612 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 12876
E20260808 02:08:00.179641 13017 webserver.cc:533] Webserver: sslize failed:
E20260808 02:08:00.179754 13017 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T02:08:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T02:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:00Z Disabled control of system clock
I20260808 02:08:00.225567 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34293
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34293
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/testchainca.cert with env {}
W20260808 02:08:00.344775 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:00.345055 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:00.345095 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:00.349215 13021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:00.349279 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:00.349295 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:00.349311 13021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:00.349372 13021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:08:00.349406 13021 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:08:00.349429 13021 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 02:08:00.354910 13021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:00.356027 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:00.357107 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:00.365885 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:00.366044 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:00.366066 13021 server_base.cc:1034] running on GCE node
W20260808 02:08:00.365899 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:00.366793 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:00.367888 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:00.369117 13021 hybrid_clock.cc:648] HybridClock initialized: now 1786154880369098 us; error 38 us; skew 500 ppm
I20260808 02:08:00.371381 13021 webserver.cc:460] Webserver started at http://127.5.209.126:41653/ using document root <none> and password file <none>
I20260808 02:08:00.372040 13021 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:00.372145 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:00.372403 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:00.374702 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "b0c16d1872cc47128c0f11f60e7b659e"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "42f0d98b5dea18df9605be62f660243f"
server_key_iv: "ea047cc2fe286531c2dc4948a150ee0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.375339 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "b0c16d1872cc47128c0f11f60e7b659e"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "42f0d98b5dea18df9605be62f660243f"
server_key_iv: "ea047cc2fe286531c2dc4948a150ee0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.379323 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:08:00.381919 13043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:00.383244 13021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:00.383404 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "b0c16d1872cc47128c0f11f60e7b659e"
format_stamp: "Formatted at 2026-08-08 02:08:00 on dist-test-slave-bw88"
server_key: "42f0d98b5dea18df9605be62f660243f"
server_key_iv: "ea047cc2fe286531c2dc4948a150ee0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:00.383530 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:00.404968 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:00.405737 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:00.405952 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:00.413379 13021 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34293
I20260808 02:08:00.413385 13123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34293 every 8 connection(s)
I20260808 02:08:00.414391 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:00.417704 13124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:00.422317 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13021
I20260808 02:08:00.422452 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:00.422788 5957 external_mini_cluster.cc:1428] Setting key 68daf3a177c032f5bc2f9448dc4a0e15
I20260808 02:08:00.423815 13124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: Bootstrap starting.
I20260808 02:08:00.426229 13124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:00.427001 13124 log.cc:824] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:00.429083 13124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: No bootstrap required, opened a new log
I20260808 02:08:00.432108 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:00.424187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.5.209.126:34293)
0808 02:08:00.424848 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:00.424864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:00.424900 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:08:00.425587 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:00.425606 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:00.426027 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:00.426353 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:00.426799 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.427831 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.428747 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.429275 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.429508 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.430416 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:00.430422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:00.430437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:00.430816 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:00.430872 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:00.430892 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:00.431013 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:00.431137 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:08:00.431398 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260808 02:08:00.432766 13124 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } }
I20260808 02:08:00.433007 13124 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:00.433060 13124 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c16d1872cc47128c0f11f60e7b659e, State: Initialized, Role: FOLLOWER
I20260808 02:08:00.433535 13124 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } }
I20260808 02:08:00.433703 13124 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:00.433769 13124 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:00.433882 13124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:00.434991 13124 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } }
I20260808 02:08:00.435032 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:08:00.435349 13124 leader_election.cc:302] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c16d1872cc47128c0f11f60e7b659e; no voters:
I20260808 02:08:00.435684 13124 leader_election.cc:288] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:00.435791 13133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:00.436033 13133 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [term 1 LEADER]: Becoming Leader. State: Replica: b0c16d1872cc47128c0f11f60e7b659e, State: Running, Role: LEADER
I20260808 02:08:00.436362 13133 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } }
I20260808 02:08:00.436980 13124 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:00.438342 13134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c16d1872cc47128c0f11f60e7b659e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } } }
I20260808 02:08:00.438477 13134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:00.438755 13135 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c16d1872cc47128c0f11f60e7b659e. Latest consensus state: current_term: 1 leader_uuid: "b0c16d1872cc47128c0f11f60e7b659e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c16d1872cc47128c0f11f60e7b659e" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34293 } } }
I20260808 02:08:00.438843 13135 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e [sys.catalog]: This master's current role is: LEADER
W20260808 02:08:00.442140 13154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:08:00.442204 13154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:08:00.442289 13148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:00.442993 13148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:00.448179 13148 catalog_manager.cc:1347] Generated new cluster ID: cbca5d86576f4004aa371bf7539c91f1
I20260808 02:08:00.448246 13148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:00.477257 13148 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:00.478278 13148 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:00.485046 13148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c16d1872cc47128c0f11f60e7b659e: Generated new TSK 0
I20260808 02:08:00.485857 13148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:08:00.565203 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:00.500379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.5.209.126:34293)
0808 02:08:00.500530 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:00.500535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:00.500699 (+ 164us) server_negotiation.cc:407] Connection header received
0808 02:08:00.500833 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:00.500836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:00.500895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:00.501009 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:08:00.501231 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.502096 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.503037 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.503396 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.503502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.503846 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:08:00.564652 (+ 60806us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:08:00.565023 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37162: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 02:08:00.565310 13131 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37162: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 02:08:00.565317 13165 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:00.500280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34293 (local address 127.0.0.1:37162)
0808 02:08:00.500613 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:00.500626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:00.500722 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:00.501013 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:00.501015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:00.501022 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:08:00.501118 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:00.501122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.502248 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.502251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:00.502895 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:00.502903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:00.503546 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:00.503550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:00.503637 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:00.503662 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:08:00.564925 (+ 61263us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:08:00.565136 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.209.126:34293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
I20260808 02:08:00.566295 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13021
2026-08-08T02:08:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (377 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:00 dist-test-slave-bw88 krb5kdc[13183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:00 dist-test-slave-bw88 krb5kdc[13183](info): set up 2 sockets
Aug 08 02:08:00 dist-test-slave-bw88 krb5kdc[13183](info): commencing operation
krb5kdc: starting...
W20260808 02:08:02.034394 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:08:02.173204 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:36005
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:36005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:02.293968 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:02.294288 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:02.294338 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:02.298883 13206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:02.298964 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:02.298990 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:02.299031 13206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:02.299057 13206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:02.305163 13206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:36005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:02.306433 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:02.307554 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:02.316931 13212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:02.317014 13213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:02.316931 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:02.317211 13206 server_base.cc:1034] running on GCE node
I20260808 02:08:02.317894 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:02.319041 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:02.320267 13206 hybrid_clock.cc:648] HybridClock initialized: now 1786154882320252 us; error 28 us; skew 500 ppm
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:02.323108 13206 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:08:02.323966 13206 webserver.cc:460] Webserver started at http://127.5.209.126:37227/ using document root <none> and password file <none>
I20260808 02:08:02.324505 13206 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:02.324563 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:02.324744 13206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:02.326861 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "ec944d85bd6a437aa86cd5232874de56"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "13c46198ffde4f9a1cea30e475d6d9aa"
server_key_iv: "c1658a702d9cfedbca36fca171b28631"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.327386 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "ec944d85bd6a437aa86cd5232874de56"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "13c46198ffde4f9a1cea30e475d6d9aa"
server_key_iv: "c1658a702d9cfedbca36fca171b28631"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.331219 13206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:08:02.333690 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.334980 13206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:08:02.335084 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "ec944d85bd6a437aa86cd5232874de56"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "13c46198ffde4f9a1cea30e475d6d9aa"
server_key_iv: "c1658a702d9cfedbca36fca171b28631"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.335175 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:02.373968 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:02.377169 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:02.377369 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:02.384858 13206 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:36005
I20260808 02:08:02.384888 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:36005 every 8 connection(s)
I20260808 02:08:02.385939 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:02.388921 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:02.391777 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13206
I20260808 02:08:02.391880 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:02.392199 5957 external_mini_cluster.cc:1428] Setting key 39ee4bb2d5f465b036c01ace5ffcf380
I20260808 02:08:02.395257 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: Bootstrap starting.
I20260808 02:08:02.397615 13301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:02.398404 13301 log.cc:824] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:02.401189 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: No bootstrap required, opened a new log
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:02.405611 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } }
I20260808 02:08:02.405920 13301 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:02.405987 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec944d85bd6a437aa86cd5232874de56, State: Initialized, Role: FOLLOWER
I20260808 02:08:02.406539 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } }
I20260808 02:08:02.406718 13301 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:02.406800 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:02.406912 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:02.408119 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } }
I20260808 02:08:02.408506 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:02.393745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.5.209.126:36005)
0808 02:08:02.394218 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:02.394233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:02.394271 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:08:02.394904 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:02.394922 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:02.395257 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:02.395646 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:02.396498 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.397575 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.398453 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.399148 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.399341 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:02.402357 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:02.402363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:02.402381 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:02.402414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:02.405016 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.405556 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.405562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.405568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.405723 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.406047 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.406051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.406053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.406585 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:02.406805 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:02.407170 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 02:08:02.407482 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260808 02:08:02.408495 13301 leader_election.cc:302] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec944d85bd6a437aa86cd5232874de56; no voters:
I20260808 02:08:02.408869 13301 leader_election.cc:288] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:02.409034 13308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:02.409307 13308 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [term 1 LEADER]: Becoming Leader. State: Replica: ec944d85bd6a437aa86cd5232874de56, State: Running, Role: LEADER
I20260808 02:08:02.410183 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:02.410710 13308 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } }
I20260808 02:08:02.412473 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec944d85bd6a437aa86cd5232874de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } } }
I20260808 02:08:02.413293 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:02.412828 13310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec944d85bd6a437aa86cd5232874de56. Latest consensus state: current_term: 1 leader_uuid: "ec944d85bd6a437aa86cd5232874de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec944d85bd6a437aa86cd5232874de56" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 36005 } } }
I20260808 02:08:02.413524 13310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56 [sys.catalog]: This master's current role is: LEADER
W20260808 02:08:02.415027 13327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:08:02.415095 13327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:08:02.415176 13328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:02.415892 13328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:02.421610 13328 catalog_manager.cc:1347] Generated new cluster ID: e68fd65963ec49578d2129c9575e4aaa
I20260808 02:08:02.421674 13328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:02.429167 13328 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:02.430346 13328 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:02.452589 13328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec944d85bd6a437aa86cd5232874de56: Generated new TSK 0
I20260808 02:08:02.453533 13328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:08:02.487666 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:02.605386 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:02.605665 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:02.605711 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:02.609817 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:02.609879 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:02.609977 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:02.615672 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:02.616796 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:02.617904 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:02.627907 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:02.628068 13336 server_base.cc:1034] running on GCE node
W20260808 02:08:02.627893 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:02.628237 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:02.628727 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:02.629554 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:02.630741 13336 hybrid_clock.cc:648] HybridClock initialized: now 1786154882630721 us; error 32 us; skew 500 ppm
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:02.637461 13336 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:08:02.638517 13336 webserver.cc:460] Webserver started at http://127.5.209.65:45771/ using document root <none> and password file <none>
I20260808 02:08:02.639134 13336 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:02.639199 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:02.639478 13336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:02.641774 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "94408347bb1c4ff9a154b9fd71de2cbe"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "cfedbb25b85280f84070db5c525880b5"
server_key_iv: "b701e5a391e694bdfd187788ba5af774"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.642381 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "94408347bb1c4ff9a154b9fd71de2cbe"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "cfedbb25b85280f84070db5c525880b5"
server_key_iv: "b701e5a391e694bdfd187788ba5af774"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.646311 13336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:02.648815 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.650141 13336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:02.650382 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "94408347bb1c4ff9a154b9fd71de2cbe"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "cfedbb25b85280f84070db5c525880b5"
server_key_iv: "b701e5a391e694bdfd187788ba5af774"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.650502 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:02.672523 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:02.675658 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:02.675881 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:02.676540 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:02.677515 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:02.677585 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.677654 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:02.677704 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.687044 13336 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:37339
I20260808 02:08:02.687069 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:37339 every 8 connection(s)
I20260808 02:08:02.688017 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:02.694770 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13336
I20260808 02:08:02.694888 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:02.695214 5957 external_mini_cluster.cc:1428] Setting key e5c7910f9278aad26a5af1767872aa9f
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:02.703231 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:02.689937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:34313 (local address 127.5.209.126:36005)
0808 02:08:02.690083 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:02.690087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:02.690965 (+ 878us) server_negotiation.cc:407] Connection header received
0808 02:08:02.692336 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:02.692338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:02.692394 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:02.692474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:02.693847 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.694395 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.695856 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.696146 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.696270 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:02.699129 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:02.699132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:02.699134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:02.699158 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:02.700753 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.701485 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.701488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.701490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.701588 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.701996 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.701998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.702000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.702181 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:02.702255 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:02.702992 (+ 737us) server_negotiation.cc:299] Negotiation successful
0808 02:08:02.703106 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:08:02.704542 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:02.690333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36005 (local address 127.5.209.65:34313)
0808 02:08:02.690806 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:02.690848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:02.692128 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:02.692702 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:02.692709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:02.693084 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:02.693683 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:02.693700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.694551 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.694554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:02.695708 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:02.695720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.696314 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.696318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:02.696496 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:02.697201 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:02.697217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:02.698933 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:02.700914 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:02.700923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:02.700943 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:02.701384 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:02.701687 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:02.701693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:02.701698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:02.701894 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:02.702297 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:02.702306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:02.702682 (+ 376us) client_negotiation.cc:769] Sending connection context
0808 02:08:02.702911 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 02:08:02.703306 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260808 02:08:02.705914 13518 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36005
I20260808 02:08:02.706197 13518 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:02.706854 13518 heartbeater.cc:511] Master 127.5.209.126:36005 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:08:02.708493 13243 ts_manager.cc:194] Registered new tserver with Master: 94408347bb1c4ff9a154b9fd71de2cbe (127.5.209.65:37339)
I20260808 02:08:02.709730 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:34313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:08:02.726572 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:02.844465 13523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:02.844746 13523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:02.844831 13523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:02.848834 13523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:02.848915 13523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:02.848999 13523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:02.854279 13523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:02.855327 13523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:02.856377 13523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:08:02.866160 13523 server_base.cc:1034] running on GCE node
W20260808 02:08:02.866106 13530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:02.866103 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:02.866127 13532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:02.866724 13523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:02.867344 13523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:02.868556 13523 hybrid_clock.cc:648] HybridClock initialized: now 1786154882868540 us; error 34 us; skew 500 ppm
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:02.871675 13523 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:08:02.872525 13523 webserver.cc:460] Webserver started at http://127.5.209.66:43771/ using document root <none> and password file <none>
I20260808 02:08:02.873111 13523 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:02.873214 13523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:02.873513 13523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:02.875551 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "9beac936e80a4db4b912ff49772ac555"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "32a4a0d3b6e605dd8e55599764eed8f7"
server_key_iv: "f5121e907f0777f71e7333ba42a407dd"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.876142 13523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "9beac936e80a4db4b912ff49772ac555"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "32a4a0d3b6e605dd8e55599764eed8f7"
server_key_iv: "f5121e907f0777f71e7333ba42a407dd"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.880025 13523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:02.882534 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.883741 13523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:02.883883 13523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "9beac936e80a4db4b912ff49772ac555"
format_stamp: "Formatted at 2026-08-08 02:08:02 on dist-test-slave-bw88"
server_key: "32a4a0d3b6e605dd8e55599764eed8f7"
server_key_iv: "f5121e907f0777f71e7333ba42a407dd"
server_key_version: "encryptionkey@0"
I20260808 02:08:02.883997 13523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:02.906155 13523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:02.909578 13523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:02.909804 13523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:02.910490 13523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:02.911393 13523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:02.911468 13523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.911537 13523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:02.911589 13523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:02.920542 13523 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:44723
I20260808 02:08:02.920562 13698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:44723 every 8 connection(s)
I20260808 02:08:02.921501 13523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:02.923621 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13523
I20260808 02:08:02.923734 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:02.924007 5957 external_mini_cluster.cc:1428] Setting key 188e8af99ccc2ff7a47f73bd4ec4f2dd
Aug 08 02:08:02 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154882, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:08:02.936772 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:02.925161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:54973 (local address 127.5.209.126:36005)
0808 02:08:02.925318 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:02.925325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:02.925363 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:08:02.926344 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:02.926347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:02.926401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:02.926495 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:02.927794 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.928317 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.929858 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.930274 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.930426 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:02.933226 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:02.933229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:02.933231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:02.933255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:02.934756 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.935402 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.935405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.935406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.935483 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:02.935807 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:02.935809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:02.935810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:02.935981 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:02.936059 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:02.936548 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 02:08:02.936647 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:08:02.937945 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:02.924063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36005 (local address 127.5.209.66:54973)
0808 02:08:02.924567 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:02.924621 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:02.926135 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:02.926672 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:02.926680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:02.927033 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:02.927613 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:02.927637 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.928461 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.928467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:02.929722 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:02.929733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:02.930433 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:02.930436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:02.930588 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:02.931192 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:02.931209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:02.933037 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:02.934881 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:02.934895 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:02.934919 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:02.935303 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:02.935585 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:02.935587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:02.935589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:02.935723 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:02.936079 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:02.936084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:02.936324 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 02:08:02.936482 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:08:02.936734 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260808 02:08:02.939282 13699 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36005
I20260808 02:08:02.939564 13699 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:02.940163 13699 heartbeater.cc:511] Master 127.5.209.126:36005 requested a full tablet report, sending...
I20260808 02:08:02.941385 13243 ts_manager.cc:194] Registered new tserver with Master: 9beac936e80a4db4b912ff49772ac555 (127.5.209.66:44723)
I20260808 02:08:02.942323 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:54973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:08:02.954743 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:03.072566 13705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:03.072820 13705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:03.072865 13705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:03.076959 13705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:03.077026 13705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:03.077121 13705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:03.082567 13705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:36005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:03.083720 13705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:03.084985 13705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:03.097605 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:03.097587 13710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:03.097865 13705 server_base.cc:1034] running on GCE node
W20260808 02:08:03.097605 13711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:03.098287 13705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:03.098963 13705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:03.100133 13705 hybrid_clock.cc:648] HybridClock initialized: now 1786154883100119 us; error 25 us; skew 500 ppm
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.103295 13705 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:08:03.104230 13705 webserver.cc:460] Webserver started at http://127.5.209.67:44069/ using document root <none> and password file <none>
I20260808 02:08:03.104868 13705 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:03.104939 13705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:03.105189 13705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:03.107348 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "391d6ad52ec0492dab028196c352473e"
format_stamp: "Formatted at 2026-08-08 02:08:03 on dist-test-slave-bw88"
server_key: "a8ee37a63194abba826cc42c85e74a02"
server_key_iv: "5eca68afb8b726722866a7596297c41c"
server_key_version: "encryptionkey@0"
I20260808 02:08:03.107940 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "391d6ad52ec0492dab028196c352473e"
format_stamp: "Formatted at 2026-08-08 02:08:03 on dist-test-slave-bw88"
server_key: "a8ee37a63194abba826cc42c85e74a02"
server_key_iv: "5eca68afb8b726722866a7596297c41c"
server_key_version: "encryptionkey@0"
I20260808 02:08:03.111960 13705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:03.114529 13727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:03.115749 13705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:03.115875 13705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "391d6ad52ec0492dab028196c352473e"
format_stamp: "Formatted at 2026-08-08 02:08:03 on dist-test-slave-bw88"
server_key: "a8ee37a63194abba826cc42c85e74a02"
server_key_iv: "5eca68afb8b726722866a7596297c41c"
server_key_version: "encryptionkey@0"
I20260808 02:08:03.115976 13705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:03.134644 13705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:03.138165 13705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:03.138367 13705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:03.139005 13705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:03.139957 13705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:03.140010 13705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:03.140080 13705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:03.140120 13705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:03.149875 13705 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:45283
I20260808 02:08:03.149909 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:45283 every 8 connection(s)
I20260808 02:08:03.150835 13705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:03.152035 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13705
I20260808 02:08:03.152174 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:03.152499 5957 external_mini_cluster.cc:1428] Setting key 82c41d8c1bbe8190a846ee06afcd6028
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:03.164616 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:03.152843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59185 (local address 127.5.209.126:36005)
0808 02:08:03.153020 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.153025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.154088 (+ 1063us) server_negotiation.cc:407] Connection header received
0808 02:08:03.155697 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.155700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.155748 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.155820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.156993 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.157595 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.158711 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.158920 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.158994 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:03.161457 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:03.161461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:03.161463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:03.161489 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:03.162837 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.163315 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.163318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.163319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.163393 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.163685 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.163687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.163688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.163859 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:03.163934 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:03.164393 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 02:08:03.164490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:08:03.165524 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:03.153307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:36005 (local address 127.5.209.67:59185)
0808 02:08:03.153953 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:03.153996 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:03.155506 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:03.155994 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:03.156000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:03.156379 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:03.156868 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:03.156882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.157707 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.157710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:03.158607 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:03.158616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.159046 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.159048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:03.159242 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:03.159782 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:03.159797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:03.161262 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:03.162944 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:03.162949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:03.162961 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:03.163229 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:03.163471 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:03.163473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:03.163475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:03.163606 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:03.163946 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:03.163951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:03.164188 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 02:08:03.164343 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 02:08:03.164587 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":166,"threads_started":1}
I20260808 02:08:03.166751 13891 heartbeater.cc:347] Connected to a master server at 127.5.209.126:36005
I20260808 02:08:03.166999 13891 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:03.167514 13891 heartbeater.cc:511] Master 127.5.209.126:36005 requested a full tablet report, sending...
I20260808 02:08:03.168542 13243 ts_manager.cc:194] Registered new tserver with Master: 391d6ad52ec0492dab028196c352473e (127.5.209.67:45283)
I20260808 02:08:03.169113 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:59185
I20260808 02:08:03.178699 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/test-admin.keytab.
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.197144 5957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 02:08:03.200157 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:03.198634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.5.209.65:37339)
0808 02:08:03.198946 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.198962 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.198986 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:08:03.199041 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.199046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.199266 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.199396 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.199960 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
W20260808 02:08:03.200493 5957 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.201644 5957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.202632 5957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:08:03.212218 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:03.203518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.5.209.65:37339)
0808 02:08:03.203645 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.203648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.203661 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:03.203753 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.203755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.203790 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.203851 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.204348 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.205130 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.205987 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.206363 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.206446 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:03.208534 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:03.208543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:03.208556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:03.208577 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:03.210341 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.210748 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.210754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.210755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.210833 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.211163 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.211167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.211169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.211423 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:03.211655 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:03.211884 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 02:08:03.212060 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 02:08:03.215941 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:03.214987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.5.209.65:37339)
0808 02:08:03.215124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.215128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.215139 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:08:03.215242 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.215244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.215276 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.215338 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.215823 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59450: BlockingRecv error: recv got EOF from 127.0.0.1:59450 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 02:08:03.216032 5957 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.216986 5957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.217941 5957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:08:03.226485 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:03.218729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.5.209.65:37339)
0808 02:08:03.218857 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.218860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.218872 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:03.218980 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.218982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.219014 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.219075 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.219529 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.220047 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.220894 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.221121 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.221212 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:03.223250 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:03.223253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:03.223255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:03.223276 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:03.224996 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.225417 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.225420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.225421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.225494 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.225736 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.225738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.225739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.225888 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:03.226001 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:03.226192 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:08:03.226329 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 02:08:03.229012 13899 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:03.228153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.5.209.65:37339)
0808 02:08:03.228286 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.228292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.228310 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:08:03.228410 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.228412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.228446 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.228508 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.228872 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59470: BlockingRecv error: recv got EOF from 127.0.0.1:59470 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 02:08:03.229060 5957 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786154850122309-5957-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.230080 5957 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:03.230927 5957 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:08:03.238946 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:03.231722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.5.209.65:37339)
0808 02:08:03.231868 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:03.231871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:03.231884 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:03.231986 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:03.231988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:03.232021 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:03.232087 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:03.232505 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.232989 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.233813 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:03.234018 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:03.234098 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:03.236137 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:03.236140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:03.236142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:03.236164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:03.237430 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.237808 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.237810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.237812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.237887 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:03.238130 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:03.238132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:03.238133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:03.238286 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:03.238403 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:03.238643 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:08:03.238792 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:08:03.240155 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13336
I20260808 02:08:03.260845 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13523
I20260808 02:08:03.272691 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13705
I20260808 02:08:03.282722 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13206
2026-08-08T02:08:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2447 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13916](info): set up 2 sockets
Aug 08 02:08:03 dist-test-slave-bw88 krb5kdc[13916](info): commencing operation
krb5kdc: starting...
W20260808 02:08:04.478065 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:04 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab.
Entry for principal oryx/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab.
I20260808 02:08:04.608394 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35645
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab
--principal=oryx/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:04.758193 13933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:04.758445 13933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:04.758518 13933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:04.762538 13933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:04.762593 13933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:04.762614 13933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:04.762632 13933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:04.762682 13933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:04.768730 13933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab
--principal=oryx/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:04.770247 13933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:04.771385 13933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:04.780123 13942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:04.780123 13940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:04.780380 13933 server_base.cc:1034] running on GCE node
W20260808 02:08:04.780154 13945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:04.781059 13933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:04.782261 13933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:04.783452 13933 hybrid_clock.cc:648] HybridClock initialized: now 1786154884783421 us; error 45 us; skew 500 ppm
Aug 08 02:08:04 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:04.786203 13933 init.cc:377] Logged in from keytab as oryx/127.5.209.126@KRBTEST.COM (short username oryx)
I20260808 02:08:04.787089 13933 webserver.cc:460] Webserver started at http://127.5.209.126:41207/ using document root <none> and password file <none>
I20260808 02:08:04.787667 13933 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:04.787760 13933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:04.787999 13933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:04.790105 13933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "795ba3cf597c4d358efae359dfcf13b6"
format_stamp: "Formatted at 2026-08-08 02:08:04 on dist-test-slave-bw88"
server_key: "f50aad3022a74bcd71ebd593dd5db544"
server_key_iv: "69387e9dd9c69fabf5051a6020f88b3a"
server_key_version: "encryptionkey@0"
I20260808 02:08:04.790699 13933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "795ba3cf597c4d358efae359dfcf13b6"
format_stamp: "Formatted at 2026-08-08 02:08:04 on dist-test-slave-bw88"
server_key: "f50aad3022a74bcd71ebd593dd5db544"
server_key_iv: "69387e9dd9c69fabf5051a6020f88b3a"
server_key_version: "encryptionkey@0"
I20260808 02:08:04.794667 13933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:04.797005 13953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:04.798213 13933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:04.798367 13933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "795ba3cf597c4d358efae359dfcf13b6"
format_stamp: "Formatted at 2026-08-08 02:08:04 on dist-test-slave-bw88"
server_key: "f50aad3022a74bcd71ebd593dd5db544"
server_key_iv: "69387e9dd9c69fabf5051a6020f88b3a"
server_key_version: "encryptionkey@0"
I20260808 02:08:04.798480 13933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:04.810921 13933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:04.813681 13933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:04.813877 13933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:04.820879 13933 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:35645
I20260808 02:08:04.821183 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:35645 every 8 connection(s)
I20260808 02:08:04.821918 13933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:04.824697 14025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:04.827682 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 13933
I20260808 02:08:04.827840 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:04.828148 5957 external_mini_cluster.cc:1428] Setting key df20871a088d61e75bc1ffb9f7779f6e
I20260808 02:08:04.831045 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Bootstrap starting.
I20260808 02:08:04.833319 14025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:04.834133 14025 log.cc:824] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:04.836357 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: No bootstrap required, opened a new log
Aug 08 02:08:04 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:04.840302 14025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } }
I20260808 02:08:04.840551 14025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:04.840639 14025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795ba3cf597c4d358efae359dfcf13b6, State: Initialized, Role: FOLLOWER
I20260808 02:08:04.841094 14025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } }
I20260808 02:08:04.841281 14025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:04.841372 14025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:04.841481 14025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:04.842486 14025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } }
I20260808 02:08:04.842634 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:04.829661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.5.209.126:35645)
0808 02:08:04.830126 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:04.830146 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:04.830182 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:08:04.830791 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:04.830807 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:04.831168 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:04.831564 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:04.832396 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:04.833619 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:04.834506 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:04.834981 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:04.835164 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:04.837878 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:04.837883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:04.837897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:04.837924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:04.839606 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:04.840064 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:04.840069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:04.840073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:04.840193 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:04.840490 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:04.840494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:04.840496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:04.841047 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:04.841282 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:04.841580 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:08:04.841879 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260808 02:08:04.842900 14025 leader_election.cc:302] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 795ba3cf597c4d358efae359dfcf13b6; no voters:
I20260808 02:08:04.843255 14025 leader_election.cc:288] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:04.843333 14035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:04.843534 14035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 LEADER]: Becoming Leader. State: Replica: 795ba3cf597c4d358efae359dfcf13b6, State: Running, Role: LEADER
I20260808 02:08:04.843859 14035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } }
I20260808 02:08:04.844717 14025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:04.845736 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795ba3cf597c4d358efae359dfcf13b6. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } }
I20260808 02:08:04.845772 14037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } }
I20260808 02:08:04.845868 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:04.845902 14037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:04.847029 14049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:04.849723 14049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:04.855652 14049 catalog_manager.cc:1347] Generated new cluster ID: 572990ab8fe046a59afc1da1d173d8ce
I20260808 02:08:04.855728 14049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:04.869030 14049 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:04.870049 14049 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:04.875958 14049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Generated new TSK 0
I20260808 02:08:04.876752 14049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
I20260808 02:08:04.922303 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:05.056044 14070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:05.056301 14070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:05.056375 14070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:05.060359 14070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:05.060410 14070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:05.060528 14070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:05.065702 14070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:05.066731 14070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:05.067771 14070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:05.077625 14075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.077565 14078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.077589 14076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.077728 14070 server_base.cc:1034] running on GCE node
I20260808 02:08:05.078270 14070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:05.079152 14070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:05.080330 14070 hybrid_clock.cc:648] HybridClock initialized: now 1786154885080294 us; error 47 us; skew 500 ppm
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:05.083287 14070 init.cc:377] Logged in from keytab as oryx/127.5.209.65@KRBTEST.COM (short username oryx)
I20260808 02:08:05.084179 14070 webserver.cc:460] Webserver started at http://127.5.209.65:42535/ using document root <none> and password file <none>
I20260808 02:08:05.084753 14070 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.084848 14070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.085211 14070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:05.088613 14070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "de5d1898dce1416ca5420bfb462323ac"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "6563444c3d5aae05c60a084b4e4873b1"
server_key_iv: "2e04787a0252b3d71030bdb9de64950c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.089419 14070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "de5d1898dce1416ca5420bfb462323ac"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "6563444c3d5aae05c60a084b4e4873b1"
server_key_iv: "2e04787a0252b3d71030bdb9de64950c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.094587 14070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:08:05.097082 14086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.098376 14070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:08:05.098548 14070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "de5d1898dce1416ca5420bfb462323ac"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "6563444c3d5aae05c60a084b4e4873b1"
server_key_iv: "2e04787a0252b3d71030bdb9de64950c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.098660 14070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:05.113821 14070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:05.117233 14070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:05.117473 14070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:05.118075 14070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:05.118974 14070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:05.119055 14070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.119129 14070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:05.119182 14070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.128043 14070 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:40919
I20260808 02:08:05.128118 14238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:40919 every 8 connection(s)
I20260808 02:08:05.128952 14070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:05.130584 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 14070
I20260808 02:08:05.130712 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:05.131006 5957 external_mini_cluster.cc:1428] Setting key 4f496e661770842fec2022616462599b
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:08:05.144789 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.131495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:42889 (local address 127.5.209.126:35645)
0808 02:08:05.131636 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.131642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.132068 (+ 426us) server_negotiation.cc:407] Connection header received
0808 02:08:05.133118 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.133121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.133164 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.133233 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.134971 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.135864 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.137737 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.137967 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.138054 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.140975 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.140984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.140986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.141009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.142564 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.143159 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.143162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.143164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.143250 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.143598 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.143614 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.143616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.143832 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.143943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.144445 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.144576 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:08:05.145725 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.131178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:42889)
0808 02:08:05.131909 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.131960 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.132889 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.133465 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.133477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.134073 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.134774 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.134800 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.136039 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.136044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.137604 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.137620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.138154 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.138161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.138402 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.139057 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.139085 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.140809 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.142698 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.142706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.142725 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.143022 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.143350 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.143353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.143355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.143500 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.143945 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.143953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.144209 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.144400 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.144729 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.66@KRBTEST.COM" created.
I20260808 02:08:05.147127 14239 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:05.147387 14239 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:05.147913 14239 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
I20260808 02:08:05.149580 13977 ts_manager.cc:194] Registered new tserver with Master: de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919)
I20260808 02:08:05.150696 13977 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:42889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
I20260808 02:08:05.165000 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:05.283375 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:05.283627 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:05.283705 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:05.287714 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:05.287774 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:05.287876 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:05.293121 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:05.294234 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:05.295300 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:05.305430 14253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.305586 14254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.305583 14259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.305946 14247 server_base.cc:1034] running on GCE node
I20260808 02:08:05.306389 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:05.307037 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:05.308241 14247 hybrid_clock.cc:648] HybridClock initialized: now 1786154885308228 us; error 27 us; skew 500 ppm
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:05.311223 14247 init.cc:377] Logged in from keytab as oryx/127.5.209.66@KRBTEST.COM (short username oryx)
I20260808 02:08:05.312134 14247 webserver.cc:460] Webserver started at http://127.5.209.66:40009/ using document root <none> and password file <none>
I20260808 02:08:05.312696 14247 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.312788 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.313031 14247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:05.315096 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "27de2be8517e4430b6535b92e7d725c4"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "47fc77c421320b4a642ff97a081fc4bf"
server_key_iv: "9adfd0018d7ffc7d7d71213733811e79"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.315675 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "27de2be8517e4430b6535b92e7d725c4"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "47fc77c421320b4a642ff97a081fc4bf"
server_key_iv: "9adfd0018d7ffc7d7d71213733811e79"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.319761 14247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:08:05.322154 14268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.323379 14247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:05.323518 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "27de2be8517e4430b6535b92e7d725c4"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "47fc77c421320b4a642ff97a081fc4bf"
server_key_iv: "9adfd0018d7ffc7d7d71213733811e79"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.323647 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:05.340363 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:05.343649 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:05.343845 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:05.344501 14247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:05.345423 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:05.345516 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.345609 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:05.345660 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.355145 14247 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:39221
I20260808 02:08:05.355218 14432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:39221 every 8 connection(s)
I20260808 02:08:05.356086 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:05.362325 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 14247
I20260808 02:08:05.362422 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:05.362721 5957 external_mini_cluster.cc:1428] Setting key 6dd65dee0b1821604e05d3502235ee95
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:05.369726 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.357930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:38681 (local address 127.5.209.126:35645)
0808 02:08:05.358070 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.358074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.359152 (+ 1078us) server_negotiation.cc:407] Connection header received
0808 02:08:05.360083 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.360086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.360154 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.360257 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.361474 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.362010 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.363337 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.363574 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.363687 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.366144 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.366147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.366149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.366172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.367622 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.368150 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.368154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.368156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.368240 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.368580 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.368584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.368586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.368775 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.368884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.369412 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.369543 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:08:05.370472 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.358312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:38681)
0808 02:08:05.359000 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.359045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.359851 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.360433 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.360439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.360785 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.361267 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.361280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.362149 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.362152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.363209 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.363219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.363726 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.363731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.363901 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.364487 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.364503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.365969 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.367758 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.367764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.367779 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.368040 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.368336 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.368340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.368342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.368482 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.368886 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.368894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.369140 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.369324 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.369681 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":119,"threads_started":1}
I20260808 02:08:05.371628 14433 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:05.371877 14433 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:05.372364 14433 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:08:05.373541 13977 ts_manager.cc:194] Registered new tserver with Master: 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:05.374174 13977 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:38681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
I20260808 02:08:05.392952 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:05.511062 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:05.511312 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:05.511389 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:05.515404 14440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:05.515458 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:05.515563 14440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:05.520843 14440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:05.521957 14440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:05.522979 14440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:05.532940 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.532994 14440 server_base.cc:1034] running on GCE node
W20260808 02:08:05.532940 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.532991 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.533619 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:05.534247 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:05.535434 14440 hybrid_clock.cc:648] HybridClock initialized: now 1786154885535398 us; error 47 us; skew 500 ppm
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:05.538276 14440 init.cc:377] Logged in from keytab as oryx/127.5.209.67@KRBTEST.COM (short username oryx)
I20260808 02:08:05.539136 14440 webserver.cc:460] Webserver started at http://127.5.209.67:34289/ using document root <none> and password file <none>
I20260808 02:08:05.539705 14440 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.539798 14440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.540036 14440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:05.542140 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "ef0e65ae24e44b8d9f5daff7e734767b"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "e144457152b9fde96b821b9572991349"
server_key_iv: "8f3c77b7a93c676827dc54cc31b8f27e"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.542706 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "ef0e65ae24e44b8d9f5daff7e734767b"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "e144457152b9fde96b821b9572991349"
server_key_iv: "8f3c77b7a93c676827dc54cc31b8f27e"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.547212 14440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:05.549716 14458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.550920 14440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:08:05.551056 14440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "ef0e65ae24e44b8d9f5daff7e734767b"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "e144457152b9fde96b821b9572991349"
server_key_iv: "8f3c77b7a93c676827dc54cc31b8f27e"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.551173 14440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:05.567415 14440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:05.570238 14440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:05.570441 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:05.571038 14440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:05.571889 14440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:05.571960 14440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.572032 14440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:05.572075 14440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.581249 14440 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:43931
I20260808 02:08:05.581329 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:43931 every 8 connection(s)
I20260808 02:08:05.582222 14440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:05.590245 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 14440
I20260808 02:08:05.590418 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:05.590785 5957 external_mini_cluster.cc:1428] Setting key cb6e6f5b7893d7c341a831bf58b33963
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:05.596361 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.584131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:44633 (local address 127.5.209.126:35645)
0808 02:08:05.584281 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.584285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.585370 (+ 1085us) server_negotiation.cc:407] Connection header received
0808 02:08:05.586261 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.586264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.586312 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.586387 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.587562 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.588083 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.589228 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.589523 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.589606 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.592583 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.592587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.592590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.592617 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.594232 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.594766 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.594770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.594772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.594860 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.595233 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.595237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.595239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.595432 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.595544 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.596063 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.596191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:08:05.597299 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.584496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:44633)
0808 02:08:05.585189 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.585230 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.586074 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.586563 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.586569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.586931 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.587434 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.587448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.588214 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.588217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.589119 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.589128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.589660 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.589663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.589809 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.590438 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.590462 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.592424 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.594358 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.594364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.594380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.594655 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.594956 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.594959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.594961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.595122 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.595544 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.595560 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.595819 (+ 259us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.596011 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.596339 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":120,"threads_started":1}
I20260808 02:08:05.598425 14618 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:05.598685 14618 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:05.599196 14618 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
I20260808 02:08:05.600247 13977 ts_manager.cc:194] Registered new tserver with Master: ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:05.600821 13977 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:44633
I20260808 02:08:05.604691 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.209.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.125.keytab.
Entry for principal oryx/127.5.209.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.125.keytab.
I20260808 02:08:05.634413 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:38817
--webserver_interface=127.5.209.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.125.keytab
--principal=oryx/127.5.209.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35645,127.5.209.125:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:05.750053 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:05.750304 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:05.750344 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:05.754276 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:05.754331 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:05.754350 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:05.754393 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:05.754426 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:05.759691 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35645,127.5.209.125:38817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.125.keytab
--principal=oryx/127.5.209.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.209.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
W20260808 02:08:05.759907 14625 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.209.126:35645,127.5.209.125:38817'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 02:08:05.760807 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:05.761920 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:05.770725 14632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.770778 14634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.770744 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.770818 14625 server_base.cc:1034] running on GCE node
I20260808 02:08:05.771960 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:05.773391 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:05.774574 14625 hybrid_clock.cc:648] HybridClock initialized: now 1786154885774536 us; error 54 us; skew 500 ppm
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:05.777458 14625 init.cc:377] Logged in from keytab as oryx/127.5.209.125@KRBTEST.COM (short username oryx)
I20260808 02:08:05.778393 14625 webserver.cc:460] Webserver started at http://127.5.209.125:45135/ using document root <none> and password file <none>
I20260808 02:08:05.778981 14625 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.779076 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.779292 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:05.781472 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/instance:
uuid: "3b0dd0822d5d44918988f8e918ba3898"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "2e9e0206963d7ffbdcd239ce65895c32"
server_key_iv: "7e3eb36e9280c06f2afedf1e23b4b44c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.782068 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal/instance:
uuid: "3b0dd0822d5d44918988f8e918ba3898"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "2e9e0206963d7ffbdcd239ce65895c32"
server_key_iv: "7e3eb36e9280c06f2afedf1e23b4b44c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.785836 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:08:05.788208 14643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.789434 14625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:05.789587 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "3b0dd0822d5d44918988f8e918ba3898"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "2e9e0206963d7ffbdcd239ce65895c32"
server_key_iv: "7e3eb36e9280c06f2afedf1e23b4b44c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.789705 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:05.821847 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:05.825202 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:05.825449 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:05.833724 14625 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.125:38817
I20260808 02:08:05.834064 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.125:38817 every 8 connection(s)
I20260808 02:08:05.834740 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
I20260808 02:08:05.837682 14716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:05.841532 14716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:08:05.841850 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 14625
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.125@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:05.855636 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.843468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.5.209.126:35645)
0808 02:08:05.843630 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.843638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.844091 (+ 453us) server_negotiation.cc:407] Connection header received
0808 02:08:05.844940 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.844943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.844999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.845107 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.846403 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.846995 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.848147 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.848373 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.848488 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.851177 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.851182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.851186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.851216 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.852999 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.853647 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.853653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.853655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.853749 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.854191 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.854194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.854196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.854509 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.854595 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.855290 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.855412 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:08:05.856632 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.843431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.5.209.125:38817)
0808 02:08:05.843877 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.843896 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.843936 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:08:05.844712 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.844731 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.844969 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.845119 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.846216 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.847294 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.848124 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.848505 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.848692 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.851720 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.851728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.851743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.851768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.853722 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.854284 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.854290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.854291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.854370 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.854770 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.854772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.854773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.855015 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.855204 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.855752 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.855886 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":3840,"thread_start_us":159,"threads_started":1}
I20260808 02:08:05.856637 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.843431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35460)
0808 02:08:05.843958 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.843997 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.844721 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.845574 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.845578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.845946 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.846261 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.846272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.847173 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.847176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.848014 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.848024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.848529 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.848533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.848686 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.849393 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.849419 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.851024 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.853121 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.853129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.853138 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.853534 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.853854 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.853857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.853859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.854069 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.854727 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.854737 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.855014 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.855204 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.855575 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":1,"spinlock_wait_cycles":49536,"thread_start_us":190,"threads_started":1}
I20260808 02:08:05.858155 14716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.125@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:05.865897 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.858703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52770)
0808 02:08:05.858835 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.858853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.858925 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.859404 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.859407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.859537 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.859636 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.859641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.860309 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.860311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.860819 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.860825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.861241 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.861243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.861315 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.861783 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.861794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.862948 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.864590 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.864593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.864594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.864773 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.865009 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.865011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.865012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.865083 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.865473 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.865475 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.865552 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.865648 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.865746 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 02:08:05.865926 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.859074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.5.209.125:38817)
0808 02:08:05.859190 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.859195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.859208 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:05.859243 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.859246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.859287 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.859362 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.859737 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.860218 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.860915 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.861134 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.861220 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.863074 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.863077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.863079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.863100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.864496 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.864856 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.864858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.864859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.864929 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.865161 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.865163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.865164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.865330 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.865463 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.865709 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.865797 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:08:05.869287 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Bootstrap starting.
I20260808 02:08:05.871958 14716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:05.872731 14716 log.cc:824] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:05.874889 14716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: No bootstrap required, opened a new log
I20260808 02:08:05.878002 14716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } }
I20260808 02:08:05.878244 14716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:05.878326 14716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0dd0822d5d44918988f8e918ba3898, State: Initialized, Role: FOLLOWER
I20260808 02:08:05.878820 14716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } }
I20260808 02:08:05.879313 14722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } } }
I20260808 02:08:05.879441 14722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:05.880025 14716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:08:05.885023 14740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:08:05.885176 14740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:08:05.887264 14625 master_runner.cc:428] Detected that this master 3b0dd0822d5d44918988f8e918ba3898 is joining an existing cluster
I20260808 02:08:05.887331 14625 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.209.125:38817
I20260808 02:08:05.898383 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.889856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.5.209.125:38817)
0808 02:08:05.889971 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.889978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.890424 (+ 446us) server_negotiation.cc:407] Connection header received
0808 02:08:05.890589 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.890592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.890632 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.890701 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.891248 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.892020 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893404 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.893730 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.895025 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.895028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.895031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.895053 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.896660 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897104 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.897108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.897111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.897195 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897510 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.897515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.897516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.897753 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.897839 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.898088 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.898235 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":33}
I20260808 02:08:05.898398 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.889749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52784)
0808 02:08:05.890332 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.890346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.890490 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.890754 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.890759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.890981 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.891132 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.891138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.892168 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.892177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.893224 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.893236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.893924 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.894034 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.894518 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.894536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.894887 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.896801 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.896811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.896813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.897006 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.897293 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.897298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.897411 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.897827 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.897830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.897913 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.898026 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.898183 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":335,"thread_start_us":111,"threads_started":1}
I20260808 02:08:05.898911 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.889854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.5.209.126:35645)
0808 02:08:05.890004 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.890009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.890263 (+ 254us) server_negotiation.cc:407] Connection header received
0808 02:08:05.890450 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.890453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.890508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.890594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.891317 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.892083 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893140 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.893460 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893559 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.895045 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.895061 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.895064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.895091 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.897006 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897568 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.897572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.897573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.897656 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897974 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.897977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.897979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.898207 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.898280 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.898633 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.898753 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:08:05.899437 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.889743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35462)
0808 02:08:05.890178 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.890194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.890342 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.890628 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.890631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.890954 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.891141 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.891151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.892229 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.892234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.893018 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.893028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.893599 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.893603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.893728 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.894323 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.894339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.894881 (+ 542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.897152 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.897158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.897439 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.897757 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.897760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.897762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.897855 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.898357 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.898360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.898449 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.899112 (+ 663us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.899252 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":79,"thread_start_us":92,"threads_started":1}
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.126@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:05.912698 14722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:05.912896 14722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } }
I20260808 02:08:05.913831 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.903956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52794)
0808 02:08:05.904368 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.904428 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.904565 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.904865 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.904872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.905315 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.905494 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.905502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.906175 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.906179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.907171 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.907182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.907607 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.907611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.907756 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.908223 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.908238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.909925 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.911961 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.911971 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.911979 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.912223 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.912549 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.912552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.912554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.912656 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.913130 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.913137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.913269 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.913451 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.913645 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
I20260808 02:08:05.913952 14722 leader_election.cc:288] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645)
I20260808 02:08:05.914296 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.904009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.5.209.125:38817)
0808 02:08:05.904155 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.904160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.904515 (+ 355us) server_negotiation.cc:407] Connection header received
0808 02:08:05.904693 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.904695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.904737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.904801 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.905605 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.906060 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.907299 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.907505 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.907574 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.910083 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.910087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.910090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.910115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.911837 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.912336 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.912341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.912342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.912431 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.912772 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.912778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.912782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.912994 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.913124 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.914035 (+ 911us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.914155 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:08:05.914711 13991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b0dd0822d5d44918988f8e918ba3898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795ba3cf597c4d358efae359dfcf13b6" is_pre_election: true
I20260808 02:08:05.915848 14647 leader_election.cc:302] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3b0dd0822d5d44918988f8e918ba3898; no voters: 795ba3cf597c4d358efae359dfcf13b6
I20260808 02:08:05.916667 13977 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.209.125:38817: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898"
member_type: NON_VOTER
last_known_addr {
host: "127.5.209.125"
port: 38817
}
attrs {
promote: true
}
}
dest_uuid: "795ba3cf597c4d358efae359dfcf13b6"
cas_config_opid_index: -1
I20260808 02:08:05.917411 13977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } }
I20260808 02:08:05.917531 14722 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:08:05.920295 14679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:05.920060 14757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3b0dd0822d5d44918988f8e918ba3898 (127.5.209.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } }
I20260808 02:08:05.921778 14679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 02:08:05.921993 14757 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.209.125:38817
I20260808 02:08:05.922860 14757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:05.923174 14625 master.cc:537] Master@127.5.209.125:38817 shutting down...
I20260808 02:08:05.923120 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } }
I20260808 02:08:05.923537 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.5.209.124@KRBTEST.COM; defaulting to no policy
W20260808 02:08:05.925684 13956 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 02:08:05.926748 13956 consensus_peers.cc:487] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 -> Peer 3b0dd0822d5d44918988f8e918ba3898 (127.5.209.125:38817): Couldn't send request to peer 3b0dd0822d5d44918988f8e918ba3898. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.124@KRBTEST.COM" created.
I20260808 02:08:05.929991 14625 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:05.930374 14625 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:05.930552 14625 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: stopping tablet replica
I20260808 02:08:05.936424 14625 master.cc:552] Master@127.5.209.125:38817 shutdown complete.
I20260808 02:08:05.936497 14625 master_runner.cc:305] Clearing existing system tablet
I20260808 02:08:05.936743 14625 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.936816 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.938063 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 02:08:05.939038 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.939329 14625 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:05.939459 14625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:05.939493 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "3b0dd0822d5d44918988f8e918ba3898"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "2e9e0206963d7ffbdcd239ce65895c32"
server_key_iv: "7e3eb36e9280c06f2afedf1e23b4b44c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.941872 14625 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:05.944943 14625 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:08:05.945043 14625 log.cc:1196] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 02:08:05.945416 14625 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Deleting consensus metadata
I20260808 02:08:05.945595 14625 master_runner.cc:315] Copying system tablet from 127.5.209.126:35645
I20260808 02:08:05.946810 14625 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: tablet copy: Beginning tablet copy session from remote peer at address 127.5.209.126:35645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.124.keytab.
Entry for principal oryx/127.5.209.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.124.keytab.
I20260808 02:08:05.949738 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:33253
--webserver_interface=127.5.209.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.124.keytab
--principal=oryx/127.5.209.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:08:05.957093 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.947523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.5.209.126:35645)
0808 02:08:05.947675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:05.947680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:05.947972 (+ 292us) server_negotiation.cc:407] Connection header received
0808 02:08:05.948185 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:05.948188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:05.948246 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:05.948333 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:05.948877 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.949742 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.951061 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.951469 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.951610 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.952919 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:05.952923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:05.952926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:05.952958 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:05.954679 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.955272 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.955278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.955283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.955387 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:05.955937 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:05.955943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:05.955947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:05.956222 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:05.956312 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:05.956779 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 02:08:05.956906 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:08:05.957177 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:05.947439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35478)
0808 02:08:05.947842 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:05.947861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:05.948029 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:05.948377 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:05.948380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:05.948593 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:05.948729 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.948737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.949921 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.949924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.950921 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:05.950932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:05.951764 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:05.951767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:05.951844 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:05.952408 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:05.952421 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:05.952758 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:05.954802 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.954808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.954814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.955119 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.955637 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:05.955643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:05.955648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:05.955809 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:05.956403 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:05.956409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:05.956541 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:08:05.956689 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:08:05.956878 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":97,"threads_started":1}
I20260808 02:08:05.958097 14006 tablet_copy_service.cc:140] P 795ba3cf597c4d358efae359dfcf13b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b0dd0822d5d44918988f8e918ba3898 ({username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:35478)
I20260808 02:08:05.958254 14006 tablet_copy_service.cc:161] P 795ba3cf597c4d358efae359dfcf13b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b0dd0822d5d44918988f8e918ba3898 at {username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:35478: session id = 3b0dd0822d5d44918988f8e918ba3898-00000000000000000000000000000000
I20260808 02:08:05.960165 14006 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:08:05.962069 14625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:08:05.964526 14625 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: tablet copy: Starting download of 0 data blocks...
I20260808 02:08:05.964725 14625 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: tablet copy: Starting download of 1 WAL segments...
I20260808 02:08:05.966681 14625 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:08:05.968514 14006 tablet_copy_service.cc:342] P 795ba3cf597c4d358efae359dfcf13b6: Request end of tablet copy session 3b0dd0822d5d44918988f8e918ba3898-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:35478
I20260808 02:08:05.968641 14006 tablet_copy_service.cc:433] P 795ba3cf597c4d358efae359dfcf13b6: ending tablet copy session 3b0dd0822d5d44918988f8e918ba3898-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b0dd0822d5d44918988f8e918ba3898
I20260808 02:08:05.977764 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:05.979552 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.979801 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:05.979843 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:05.980173 14625 server_base.cc:1034] running on GCE node
I20260808 02:08:05.980340 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:05.980628 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:05.981768 14625 hybrid_clock.cc:648] HybridClock initialized: now 1786154885981753 us; error 30 us; skew 500 ppm
Aug 08 02:08:05 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154885, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:05.983450 14625 init.cc:377] Logged in from keytab as oryx/127.5.209.125@KRBTEST.COM (short username oryx)
I20260808 02:08:05.983919 14625 webserver.cc:460] Webserver started at http://127.5.209.125:35895/ using document root <none> and password file <none>
I20260808 02:08:05.984123 14625 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:05.984169 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:05.985492 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:05.986541 14793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:05.986828 14625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:05.986907 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "3b0dd0822d5d44918988f8e918ba3898"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "2e9e0206963d7ffbdcd239ce65895c32"
server_key_iv: "7e3eb36e9280c06f2afedf1e23b4b44c"
server_key_version: "encryptionkey@0"
I20260808 02:08:05.986989 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.023922 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:06.026427 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:06.032111 14625 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.125:38817
I20260808 02:08:06.032166 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.125:38817 every 8 connection(s)
I20260808 02:08:06.032909 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
I20260808 02:08:06.035064 14862 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:08:06.036274 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Bootstrap starting.
I20260808 02:08:06.047585 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:08:06.047979 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Bootstrap complete.
I20260808 02:08:06.048719 14862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } }
I20260808 02:08:06.048892 14862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b0dd0822d5d44918988f8e918ba3898, State: Initialized, Role: LEARNER
I20260808 02:08:06.049077 14862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } }
I20260808 02:08:06.049549 14866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } }
I20260808 02:08:06.049687 14862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:06.049729 14866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:08:06.053058 14881 catalog_manager.cc:1259] Loaded cluster ID: 572990ab8fe046a59afc1da1d173d8ce
I20260808 02:08:06.053184 14881 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: loading cluster ID for follower catalog manager: success
I20260808 02:08:06.055742 14881 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: acquiring CA information for follower catalog manager: success
I20260808 02:08:06.057230 14881 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:08:06.091092 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:06.091382 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:06.091463 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:06.095561 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:06.095642 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:06.095681 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:06.095718 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:06.095753 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:06.101239 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.124.keytab
--principal=oryx/127.5.209.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.209.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:06.102622 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:06.103789 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:06.112933 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.113013 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.113021 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.113413 14775 server_base.cc:1034] running on GCE node
I20260808 02:08:06.113932 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:06.115012 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:06.116212 14775 hybrid_clock.cc:648] HybridClock initialized: now 1786154886116199 us; error 31 us; skew 500 ppm
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:06.119676 14775 init.cc:377] Logged in from keytab as oryx/127.5.209.124@KRBTEST.COM (short username oryx)
I20260808 02:08:06.120580 14775 webserver.cc:460] Webserver started at http://127.5.209.124:39637/ using document root <none> and password file <none>
I20260808 02:08:06.121181 14775 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.121279 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.121554 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:06.123776 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/instance:
uuid: "19f3be198c5247c994d327ca678ae2f4"
format_stamp: "Formatted at 2026-08-08 02:08:06 on dist-test-slave-bw88"
server_key: "6bc826cf3b1243bc4254b4cae4074c5d"
server_key_iv: "d1332dbe6cbad02eb6b5eac0a901ceee"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.124397 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal/instance:
uuid: "19f3be198c5247c994d327ca678ae2f4"
format_stamp: "Formatted at 2026-08-08 02:08:06 on dist-test-slave-bw88"
server_key: "6bc826cf3b1243bc4254b4cae4074c5d"
server_key_iv: "d1332dbe6cbad02eb6b5eac0a901ceee"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.128626 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 02:08:06.131757 14900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.133102 14775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:06.133200 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "19f3be198c5247c994d327ca678ae2f4"
format_stamp: "Formatted at 2026-08-08 02:08:06 on dist-test-slave-bw88"
server_key: "6bc826cf3b1243bc4254b4cae4074c5d"
server_key_iv: "d1332dbe6cbad02eb6b5eac0a901ceee"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.133279 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.150823 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:06.153061 14239 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
I20260808 02:08:06.154466 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:06.154693 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:06.167126 14775 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.124:33253
I20260808 02:08:06.167584 14972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.124:33253 every 8 connection(s)
I20260808 02:08:06.168284 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
I20260808 02:08:06.169536 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 14775
I20260808 02:08:06.172101 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:06.176817 14974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.124@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:06.187671 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.171310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.5.209.124:33253)
0808 02:08:06.171921 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.171949 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.171996 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:08:06.172982 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.173003 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.173423 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.173799 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.174711 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.175982 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.176890 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.177527 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.177723 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.181758 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.181763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.181772 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.181823 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.184319 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.184813 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.184819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.184826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.185025 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.185418 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.185421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.185423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.185852 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.186167 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.186376 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.186661 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":166,"threads_started":1}
I20260808 02:08:06.189839 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.179031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.5.209.126:35645)
0808 02:08:06.179210 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.179216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.179572 (+ 356us) server_negotiation.cc:407] Connection header received
0808 02:08:06.179766 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.179768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.179812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.179898 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.181092 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.181740 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.183296 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.183513 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.183599 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.186257 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.186260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.186262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.186287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.187919 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.188443 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.188446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.188448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.188537 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.188849 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.188852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.188854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.189033 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.189151 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.189512 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.189635 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:08:06.189988 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.178806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35482)
0808 02:08:06.179449 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.179489 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.179645 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.179955 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.179961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.180445 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.180946 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.180957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.181992 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.181998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.183126 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.183161 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.183642 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.183646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.183803 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.184330 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.184345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.186108 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.188030 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.188041 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.188046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.188275 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.188654 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.188657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.188659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.188742 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.189153 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.189161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.189277 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.189454 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.189636 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
I20260808 02:08:06.191031 14974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.124@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:06.199826 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.191515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52804)
0808 02:08:06.191636 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.191649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.191724 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.192205 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.192208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.192374 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.192535 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.192541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.193532 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.193536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.194071 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.194079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.194551 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.194555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.194676 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.195157 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.195168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.196433 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.198327 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.198330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.198332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.198523 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.198821 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.198824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.198826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.198910 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.199304 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.199307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.199407 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.199513 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.199661 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:08:06.199855 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.191639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.5.209.125:38817)
0808 02:08:06.191976 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.191980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.191991 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:08:06.192039 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.192041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.192081 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.192190 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.192670 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.193423 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.194194 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.194438 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.194551 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.196580 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.196583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.196585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.196611 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.198195 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.198635 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.198638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.198640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.198723 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.199004 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.199007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.199009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.199200 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.199313 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.199572 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.199693 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":148,"threads_started":1}
I20260808 02:08:06.200629 14974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.124@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
I20260808 02:08:06.209151 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.201071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56156)
0808 02:08:06.201192 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.201233 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.201308 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.201623 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.201625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.201790 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.201879 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.201884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.203522 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.203526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.204012 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.204019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.204502 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.204505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.204580 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.204970 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.204981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.206132 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.207763 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.207765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.207766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.207938 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.208176 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.208178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.208180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.208250 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.208691 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.208694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.208768 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.208869 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.208994 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 02:08:06.209170 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.201182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.5.209.124:33253)
0808 02:08:06.201294 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.201298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.201310 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:06.201438 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.201456 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.201501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.201597 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.201986 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.203372 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.204142 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.204392 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.204501 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.206256 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.206259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.206260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.206282 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.207652 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.208020 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.208022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.208023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.208097 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.208328 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.208330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.208331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.208570 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.208667 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.208913 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.209052 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":50}
I20260808 02:08:06.212528 14974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Bootstrap starting.
I20260808 02:08:06.215273 14974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:06.216210 14974 log.cc:824] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:06.218382 14974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: No bootstrap required, opened a new log
I20260808 02:08:06.222250 14974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } }
I20260808 02:08:06.222443 14974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:06.222478 14974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f3be198c5247c994d327ca678ae2f4, State: Initialized, Role: FOLLOWER
I20260808 02:08:06.222851 14974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } }
I20260808 02:08:06.223435 14982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } } }
I20260808 02:08:06.223616 14982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:06.224124 14974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:08:06.230345 14994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:08:06.230441 14994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:08:06.234097 14775 master_runner.cc:428] Detected that this master 19f3be198c5247c994d327ca678ae2f4 is joining an existing cluster
I20260808 02:08:06.234161 14775 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.209.124:33253
I20260808 02:08:06.241602 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 14070
I20260808 02:08:06.248515 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.237166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.5.209.125:38817)
0808 02:08:06.237311 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.237317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.238182 (+ 865us) server_negotiation.cc:407] Connection header received
0808 02:08:06.238426 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.238430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.238487 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.238544 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.239562 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.240469 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.242028 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.242432 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.242519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.244327 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.244332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.244334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.244363 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.246394 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.246965 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.246969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.246971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.247092 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.247439 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.247443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.247445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.247708 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.247786 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.248192 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.248331 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:08:06.249194 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.236827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52812)
0808 02:08:06.238044 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.238106 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.238257 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.238717 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.238720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.239060 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.239404 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.239416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.240626 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.240630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.241850 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.241867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.242737 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.242741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.242852 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.243609 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.243626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.244158 (+ 532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.246542 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.246545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.246547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.246840 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.247216 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.247219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.247221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.247317 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.247868 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.247871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.247977 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.248674 (+ 697us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.248831 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":68,"thread_start_us":313,"threads_started":1}
I20260808 02:08:06.258843 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.237651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56166)
0808 02:08:06.238100 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.238115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.238226 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.238517 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.238519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.238984 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.239393 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.239400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.251276 (+ 11876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.251280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.252446 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.252458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.253156 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.253160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.253302 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.254030 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.254045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.254603 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.256916 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.256919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.256921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.257174 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.257542 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.257546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.257548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.257737 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.258286 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.258289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.258390 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.258501 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.258635 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":60,"thread_start_us":190,"threads_started":1}
I20260808 02:08:06.259049 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.237651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.5.209.124:33253)
0808 02:08:06.237785 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.237790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.238183 (+ 393us) server_negotiation.cc:407] Connection header received
0808 02:08:06.238313 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.238318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.238351 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.238394 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.239506 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.240310 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.252577 (+ 12267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.252908 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.252996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.254758 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.254765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.254771 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.254809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.256792 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.257295 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.257301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.257306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.257431 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.257848 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.257854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.257858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.258172 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.258282 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.258627 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.258763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:08:06.261272 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:40919
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:35645
--webserver_port=42535
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:08:06.282634 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.236785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.5.209.126:35645)
0808 02:08:06.236954 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.236960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.237988 (+ 1028us) server_negotiation.cc:407] Connection header received
0808 02:08:06.238206 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.238210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.238269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.238329 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.239537 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.240310 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.252623 (+ 12313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.253042 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.253142 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.278818 (+ 25676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.278823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.278827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.278862 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.280621 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.281188 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.281192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.281194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.281293 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.281636 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.281640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.281642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.281897 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.282002 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.282335 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.282469 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:08:06.282711 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.236758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35492)
0808 02:08:06.237895 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.237911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.238070 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.238501 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.238505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.238984 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.239369 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.239381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.240468 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.240471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.241622 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.241636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.253327 (+ 11691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.253355 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.253519 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.278051 (+ 24532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.278070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.278583 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.280765 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.280769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.280772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.281047 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.281419 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.281422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.281424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.281520 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.282008 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.282011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.282135 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.282240 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.282406 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":71,"thread_start_us":313,"threads_started":1}
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154884, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.126@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
I20260808 02:08:06.296777 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.287185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56182)
0808 02:08:06.287360 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.287374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.287449 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.287889 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.287892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.288125 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.288241 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.288248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.289245 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.289249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.290128 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.290140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.290736 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.290740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.290839 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.291317 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.291333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.292947 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.295041 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.295045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.295047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.295314 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.295647 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.295650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.295652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.295750 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.296211 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.296215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.296320 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.296433 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.296586 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 02:08:06.296803 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.287302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.5.209.124:33253)
0808 02:08:06.287628 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.287633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.287646 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:06.287709 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.287712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.287762 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.287839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.288364 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.289109 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.290261 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.290609 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.290719 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.293106 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.293111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.293113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.293141 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.294903 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.295430 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.295434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.295436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.295540 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.295850 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.295853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.295855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.296104 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.296206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.296514 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.296637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260808 02:08:06.298566 13977 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.209.124:33253: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "19f3be198c5247c994d327ca678ae2f4"
member_type: NON_VOTER
last_known_addr {
host: "127.5.209.124"
port: 33253
}
attrs {
promote: true
}
}
dest_uuid: "795ba3cf597c4d358efae359dfcf13b6"
cas_config_opid_index: 5
I20260808 02:08:06.299289 13977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } }
I20260808 02:08:06.300580 14757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.302033 14934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:06.302377 14757 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.209.124:33253
I20260808 02:08:06.302589 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.302691 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:06.303797 14934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:08:06.304527 14775 master.cc:537] Master@127.5.209.124:33253 shutting down...
I20260808 02:08:06.304517 14758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:08:06.307716 13959 consensus_peers.cc:487] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 -> Peer 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124:33253): Couldn't send request to peer 19f3be198c5247c994d327ca678ae2f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:08:06.310947 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.300819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.5.209.125:38817)
0808 02:08:06.300951 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.300956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.305367 (+ 4411us) server_negotiation.cc:407] Connection header received
0808 02:08:06.305617 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.305621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.305691 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.305759 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:06.306202 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.307006 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.309288 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.310051 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.310139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.310307 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.310701 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.310794 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:08:06.311372 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.300567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52826)
0808 02:08:06.305246 (+ 4679us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.305264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.305480 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.305944 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.305947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.305966 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:06.306062 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.306068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.307156 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.307160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.309132 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.309145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.310414 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.310417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.310518 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.310572 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.311160 (+ 588us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.311251 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4586}
I20260808 02:08:06.311960 14824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 LEARNER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:08:06.312345 14775 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:06.312562 14758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 02:08:06.312932 14775 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:06.313108 14775 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: stopping tablet replica
I20260808 02:08:06.314716 14824 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.317184 14866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795ba3cf597c4d358efae359dfcf13b6. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.317353 14866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:08:06.317483 14866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: NON_VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: true } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.317562 14866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:08:06.318197 14759 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: attempting to promote NON_VOTER 3b0dd0822d5d44918988f8e918ba3898 to VOTER
I20260808 02:08:06.318652 14759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } }
I20260808 02:08:06.320019 14824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 LEARNER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:08:06.320062 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.319751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56192)
0808 02:08:06.319905 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.319978 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 02:08:06.320425 14759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 02:08:06.321059 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.320769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56204)
0808 02:08:06.320921 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.320977 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:33253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 02:08:06.321327 13959 consensus_peers.cc:487] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 -> Peer 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124:33253): Couldn't send request to peer 19f3be198c5247c994d327ca678ae2f4. Status: Network error: Client connection negotiation failed: client connection to 127.5.209.124:33253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:08:06.322154 14759 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b0dd0822d5d44918988f8e918ba3898 (127.5.209.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.323709 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.323822 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:06.323974 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.324067 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:06.324576 14824 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b0dd0822d5d44918988f8e918ba3898 (127.5.209.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.326476 15008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.326601 15008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:06.329802 14775 master.cc:552] Master@127.5.209.124:33253 shutdown complete.
I20260808 02:08:06.330124 14775 master_runner.cc:305] Clearing existing system tablet
I20260808 02:08:06.330438 14775 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.330551 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.331916 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:06.332918 15010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.333197 14775 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.333326 14775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:06.333391 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "19f3be198c5247c994d327ca678ae2f4"
format_stamp: "Formatted at 2026-08-08 02:08:06 on dist-test-slave-bw88"
server_key: "6bc826cf3b1243bc4254b4cae4074c5d"
server_key_iv: "d1332dbe6cbad02eb6b5eac0a901ceee"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.335562 14775 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:06.337960 14775 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:08:06.338053 14775 log.cc:1196] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 02:08:06.338305 14775 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Deleting consensus metadata
I20260808 02:08:06.338475 14775 master_runner.cc:315] Copying system tablet from 127.5.209.126:35645
I20260808 02:08:06.339645 14775 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.209.126:35645
I20260808 02:08:06.350314 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.340308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:35508)
0808 02:08:06.341640 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.341656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.341832 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.342155 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.342158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.342384 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.342538 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.342545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.343618 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.343622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.344495 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.344507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.345190 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.345194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.345299 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.345946 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.345962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.346521 (+ 559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.348583 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.348586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.348589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.348847 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.349172 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.349175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.349177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.349277 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.349762 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.349765 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.349873 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.349976 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.350119 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":98,"threads_started":1}
I20260808 02:08:06.350385 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.340308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.5.209.126:35645)
0808 02:08:06.340441 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.340446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.341743 (+ 1297us) server_negotiation.cc:407] Connection header received
0808 02:08:06.341962 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.341966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.342022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.342115 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.342667 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.343486 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.344671 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.345036 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.345137 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.346673 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.346677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.346679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.346708 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.348458 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.348964 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.348968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.348970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.349064 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.349415 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.349422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.349425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.349646 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.349745 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.350078 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.350223 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:08:06.351276 14006 tablet_copy_service.cc:140] P 795ba3cf597c4d358efae359dfcf13b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 19f3be198c5247c994d327ca678ae2f4 ({username='oryx', principal='oryx/127.5.209.124@KRBTEST.COM'} at 127.0.0.1:35508)
I20260808 02:08:06.351382 14006 tablet_copy_service.cc:161] P 795ba3cf597c4d358efae359dfcf13b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 19f3be198c5247c994d327ca678ae2f4 at {username='oryx', principal='oryx/127.5.209.124@KRBTEST.COM'} at 127.0.0.1:35508: session id = 19f3be198c5247c994d327ca678ae2f4-00000000000000000000000000000000
I20260808 02:08:06.352618 14006 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:08:06.354106 14775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:08:06.356666 14775 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: tablet copy: Starting download of 0 data blocks...
I20260808 02:08:06.356850 14775 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: tablet copy: Starting download of 1 WAL segments...
I20260808 02:08:06.358479 14775 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:08:06.360140 14006 tablet_copy_service.cc:342] P 795ba3cf597c4d358efae359dfcf13b6: Request end of tablet copy session 19f3be198c5247c994d327ca678ae2f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.209.124@KRBTEST.COM'} at 127.0.0.1:35508
I20260808 02:08:06.360212 14006 tablet_copy_service.cc:433] P 795ba3cf597c4d358efae359dfcf13b6: ending tablet copy session 19f3be198c5247c994d327ca678ae2f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 19f3be198c5247c994d327ca678ae2f4
I20260808 02:08:06.368175 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:06.370040 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.370043 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.370779 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.374388 14775 server_base.cc:1034] running on GCE node
I20260808 02:08:06.374631 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:06.374969 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:06.376143 14775 hybrid_clock.cc:648] HybridClock initialized: now 1786154886376129 us; error 30 us; skew 500 ppm
I20260808 02:08:06.376303 14433 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:06.377915 14775 init.cc:377] Logged in from keytab as oryx/127.5.209.124@KRBTEST.COM (short username oryx)
I20260808 02:08:06.378434 14775 webserver.cc:460] Webserver started at http://127.5.209.124:42601/ using document root <none> and password file <none>
I20260808 02:08:06.378634 14775 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.378674 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.380089 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:06.381218 15034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.381603 14775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:06.381688 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "19f3be198c5247c994d327ca678ae2f4"
format_stamp: "Formatted at 2026-08-08 02:08:06 on dist-test-slave-bw88"
server_key: "6bc826cf3b1243bc4254b4cae4074c5d"
server_key_iv: "d1332dbe6cbad02eb6b5eac0a901ceee"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.381780 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.392096 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:06.394480 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:06.399885 14775 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.124:33253
I20260808 02:08:06.400014 15105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.124:33253 every 8 connection(s)
I20260808 02:08:06.400484 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
I20260808 02:08:06.403074 15106 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:08:06.404178 15106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Bootstrap starting.
I20260808 02:08:06.417970 15106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:08:06.418323 15106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: Bootstrap complete.
I20260808 02:08:06.419054 15106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } }
I20260808 02:08:06.419245 15106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19f3be198c5247c994d327ca678ae2f4, State: Initialized, Role: LEARNER
I20260808 02:08:06.419395 15106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } }
I20260808 02:08:06.419866 15115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.420002 15106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:06.420120 15115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:08:06.423564 15129 catalog_manager.cc:1259] Loaded cluster ID: 572990ab8fe046a59afc1da1d173d8ce
I20260808 02:08:06.423679 15129 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: loading cluster ID for follower catalog manager: success
W20260808 02:08:06.424453 15005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:06.424713 15005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:06.424791 15005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 02:08:06.426229 15129 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: acquiring CA information for follower catalog manager: success
I20260808 02:08:06.427911 15129 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:08:06.429291 15005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:06.429388 15005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:06.429512 15005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:06.435070 15005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:40919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=42535
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:06.436108 15005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:06.437157 15005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:06.449232 15134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.449293 15133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.449447 15005 server_base.cc:1034] running on GCE node
W20260808 02:08:06.449546 15136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.449950 15005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:06.450704 15005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:06.451897 15005 hybrid_clock.cc:648] HybridClock initialized: now 1786154886451864 us; error 47 us; skew 500 ppm
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:06.455559 15005 init.cc:377] Logged in from keytab as oryx/127.5.209.65@KRBTEST.COM (short username oryx)
I20260808 02:08:06.456502 15005 webserver.cc:460] Webserver started at http://127.5.209.65:42535/ using document root <none> and password file <none>
I20260808 02:08:06.457090 15005 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.457219 15005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.461645 15005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:08:06.463986 15148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.465310 15005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:06.465508 15005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "de5d1898dce1416ca5420bfb462323ac"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "6563444c3d5aae05c60a084b4e4873b1"
server_key_iv: "2e04787a0252b3d71030bdb9de64950c"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.466142 15005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.491830 15005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:06.496199 15005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:06.496493 15005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:06.497423 15005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:06.498517 15005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:06.498591 15005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.498668 15005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:06.498713 15005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.518086 15005 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:40919
I20260808 02:08:06.518275 15306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:40919 every 8 connection(s)
I20260808 02:08:06.519263 15005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:06.525831 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 15005
I20260808 02:08:06.526054 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 14247
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
I20260808 02:08:06.543447 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.522193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:51933 (local address 127.5.209.124:33253)
0808 02:08:06.522702 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.522707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.523939 (+ 1232us) server_negotiation.cc:407] Connection header received
0808 02:08:06.524544 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.524548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.524615 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.524681 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.528648 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.529618 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.531258 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.531606 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.531677 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.537459 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.537466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.537472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.537509 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.539809 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.540803 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.540811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.540816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.540927 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.541426 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.541430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.541432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.541712 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.541827 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.542831 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.543057 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":92,"threads_started":1}
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:06.545506 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.522207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.5.209.65:51933)
0808 02:08:06.523835 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.523846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.524291 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.524980 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.524982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.525813 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.528458 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.528476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.529820 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.529824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.531112 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.531123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.531885 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.531888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.532161 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.534945 (+ 2784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.534978 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.537198 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.539949 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.539956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.539975 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.540668 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.541043 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.541046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.541047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.541263 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.541922 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.541930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.542347 (+ 417us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.543684 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.544136 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":2055,"spinlock_wait_cycles":1792,"thread_start_us":187,"threads_started":1}
I20260808 02:08:06.549296 15307 heartbeater.cc:347] Connected to a master server at 127.5.209.124:33253
I20260808 02:08:06.549633 15307 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.549878 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.522049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:55499 (local address 127.5.209.126:35645)
0808 02:08:06.522509 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.522516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.523617 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 02:08:06.524584 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.524587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.524639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.524700 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.538538 (+ 13838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.539412 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.540896 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.541289 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.541412 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.546044 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.546060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.546063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.546097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.547932 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.548414 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.548418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.548420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.548514 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.548835 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.548839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.548841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.549045 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.549175 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.549544 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.549718 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260808 02:08:06.550405 15307 heartbeater.cc:511] Master 127.5.209.124:33253 requested a full tablet report, sending...
I20260808 02:08:06.550638 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.522207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:55499)
0808 02:08:06.523474 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.523487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.524291 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.524993 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.524996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.525813 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.538327 (+ 12514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.538343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.539603 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.539607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.540750 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.540761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.541595 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.541599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.541782 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.543729 (+ 1947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.543745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.545852 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.548060 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.548066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.548072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.548295 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.548624 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.548627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.548630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.548724 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.549164 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.549168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.549281 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.550069 (+ 788us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.550234 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":131,"spinlock_wait_cycles":35072,"thread_start_us":203,"threads_started":1}
I20260808 02:08:06.551412 15308 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:06.551512 15308 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.551666 15308 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
I20260808 02:08:06.553126 13977 ts_manager.cc:194] Re-registered known tserver with Master: de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919)
I20260808 02:08:06.554113 13977 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:55499
I20260808 02:08:06.555816 15059 ts_manager.cc:194] Registered new tserver with Master: de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919)
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:06.563908 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:39221
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:35645
--webserver_port=40009
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:08:06.566722 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.521618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:51311 (local address 127.5.209.125:38817)
0808 02:08:06.521797 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.521802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.523244 (+ 1442us) server_negotiation.cc:407] Connection header received
0808 02:08:06.524557 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.524561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.524635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.524727 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.557068 (+ 32341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.558133 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.559092 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.559606 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.559745 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.563008 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.563015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.563022 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.563074 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.564677 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.565163 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.565167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.565168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.565253 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.565793 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.565796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.565798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.565990 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.566124 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.566417 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.566549 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:08:06.566745 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.522206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.5.209.65:51311)
0808 02:08:06.523079 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.523136 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.524291 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.524960 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.524971 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.525814 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.556665 (+ 30851us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x3476de8. stack: 00007f770c350c78 00007f770c3511a9 00007f770bf131d7 00000000005984a6 00000000005984f9 000000000059c488 00007f770c34d2ab 00007f770c34cad0 00007f77151280a5 00007f771512858c 00007f770cdce1fc 00007f77109a3b1f 00007f77109e4832 00007f77109e568c 00007f77109fb0d3 00007f77109fe54b
0808 02:08:06.556878 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.556889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.558373 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.558377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.558947 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.558956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.559724 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.559727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.559852 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.560493 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.560504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.562846 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.564805 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.564811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.564813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.565035 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.565510 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.565517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.565523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.565663 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.566113 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.566117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.566225 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.566338 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.566518 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":22,"spinlock_wait_cycles":66392832,"thread_start_us":283,"threads_started":1}
I20260808 02:08:06.567476 15309 heartbeater.cc:347] Connected to a master server at 127.5.209.125:38817
I20260808 02:08:06.567602 15309 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.567853 15309 heartbeater.cc:511] Master 127.5.209.125:38817 requested a full tablet report, sending...
I20260808 02:08:06.569231 14812 ts_manager.cc:194] Registered new tserver with Master: de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919)
I20260808 02:08:06.603005 14618 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
W20260808 02:08:06.683090 15317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:06.683372 15317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:06.683445 15317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:06.687543 15317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:06.687619 15317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:06.687708 15317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:06.693329 15317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:39221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=40009
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:06.694475 15317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:06.695618 15317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:06.706153 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.706175 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.706175 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.706532 15317 server_base.cc:1034] running on GCE node
I20260808 02:08:06.707073 15317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:06.707744 15317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:06.708913 15317 hybrid_clock.cc:648] HybridClock initialized: now 1786154886708896 us; error 27 us; skew 500 ppm
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:06.711990 15317 init.cc:377] Logged in from keytab as oryx/127.5.209.66@KRBTEST.COM (short username oryx)
I20260808 02:08:06.712895 15317 webserver.cc:460] Webserver started at http://127.5.209.66:40009/ using document root <none> and password file <none>
I20260808 02:08:06.713617 15317 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.713740 15317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.717939 15317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:08:06.720109 15339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.721634 15317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:08:06.721813 15317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "27de2be8517e4430b6535b92e7d725c4"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "47fc77c421320b4a642ff97a081fc4bf"
server_key_iv: "9adfd0018d7ffc7d7d71213733811e79"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.722462 15317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.736311 15317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:06.739377 15317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:06.739620 15317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:06.740291 15317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:06.741298 15317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:06.741410 15317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.741515 15317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:06.741552 15317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.753234 15317 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:39221
I20260808 02:08:06.753253 15491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:39221 every 8 connection(s)
I20260808 02:08:06.754565 15317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:06.761978 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 15317
I20260808 02:08:06.762225 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 14440
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:06.775791 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.758175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:53387 (local address 127.5.209.125:38817)
0808 02:08:06.758498 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.758503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.758519 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:06.759249 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.759253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.759314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.759373 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.761071 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.761894 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765332 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.765714 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765792 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.769975 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.769981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.769987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.770024 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.772790 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.773641 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.773645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.773647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.773749 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.774258 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.774262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.774264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.774543 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.774632 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.775419 (+ 787us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.775567 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260808 02:08:06.776786 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.758330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:58777 (local address 127.5.209.124:33253)
0808 02:08:06.758458 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.758463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.758702 (+ 239us) server_negotiation.cc:407] Connection header received
0808 02:08:06.759250 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.759254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.759314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.759374 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.761079 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.761816 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.764068 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.764608 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.764723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.770079 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.770087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.770094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.770139 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.773229 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.773902 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.773909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.773915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.774019 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.774400 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.774407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.774412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.774982 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.775115 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.775802 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.776505 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:08:06.778038 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.757110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.5.209.66:53387)
0808 02:08:06.757712 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.757768 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.759010 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.759562 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.759570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.760266 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.760879 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.760895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.762203 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.762205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.765188 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.765199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.765972 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.766125 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.766755 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.766772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.769790 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.772944 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.772953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.772975 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.773508 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.773874 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.773880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.773885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.774117 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.774710 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.774719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.775114 (+ 395us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.775991 (+ 877us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.776439 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1705,"spinlock_wait_cycles":63616,"thread_start_us":225,"threads_started":1}
I20260808 02:08:06.778885 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.757696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.5.209.66:58777)
0808 02:08:06.758613 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.758623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.759010 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.759562 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.759570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.760266 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.760886 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.760895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.762119 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.762123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.763908 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.763923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.765144 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.765698 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.766659 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.766691 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.769848 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.773415 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.773422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.773428 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.773768 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.774165 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.774171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.774176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.774279 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.775274 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.775279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.775448 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.777574 (+ 2126us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.777824 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":121,"spinlock_wait_cycles":65536,"thread_start_us":174,"threads_started":1}
I20260808 02:08:06.779871 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.756721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:53027 (local address 127.5.209.126:35645)
0808 02:08:06.756898 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.756904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.758435 (+ 1531us) server_negotiation.cc:407] Connection header received
0808 02:08:06.759217 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.759220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.759279 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.759378 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:06.761025 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.761894 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765172 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.765524 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.775308 (+ 9680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:06.775315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:06.775320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:06.775357 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:06.777206 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.777756 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.777759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.777761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.777865 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:06.778344 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:06.778347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:06.778349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:06.778587 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:06.778686 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.779579 (+ 893us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.779727 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:08:06.780279 15495 heartbeater.cc:347] Connected to a master server at 127.5.209.125:38817
I20260808 02:08:06.780562 15495 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.780766 15492 heartbeater.cc:347] Connected to a master server at 127.5.209.124:33253
I20260808 02:08:06.780858 15492 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.781282 15495 heartbeater.cc:511] Master 127.5.209.125:38817 requested a full tablet report, sending...
I20260808 02:08:06.781279 15492 heartbeater.cc:511] Master 127.5.209.124:33253 requested a full tablet report, sending...
I20260808 02:08:06.781692 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.757170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:53027)
0808 02:08:06.758333 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.758344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.759005 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.759561 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.759567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.760228 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:06.760876 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.760893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.762198 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.762202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.764202 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.764219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.765756 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.765762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.765898 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.766875 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:06.766892 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:06.775114 (+ 8222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:06.777376 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.777379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.777381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.777625 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.777997 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:06.778000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:06.778002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:06.778225 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:06.778961 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:06.778963 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:06.779379 (+ 416us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.781082 (+ 1703us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.781249 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":601,"spinlock_wait_cycles":32128,"thread_start_us":237,"threads_started":1}
I20260808 02:08:06.782655 15493 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:06.782752 15493 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:06.782902 15059 ts_manager.cc:194] Registered new tserver with Master: 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:06.782928 15493 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
I20260808 02:08:06.783021 14812 ts_manager.cc:194] Registered new tserver with Master: 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:06.783635 13977 ts_manager.cc:194] Re-registered known tserver with Master: 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:06.784229 13977 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:53027
I20260808 02:08:06.788746 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:43931
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:35645
--webserver_port=34289
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:06.918177 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:06.918452 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:06.918535 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:06.922950 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:06.923020 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:06.923107 15500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:06.928363 15500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:43931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=34289
--tserver_master_addrs=127.5.209.126:35645,127.5.209.125:38817,127.5.209.124:33253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:06.929466 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:06.930505 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:06.941059 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:06.941208 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.941378 15500 server_base.cc:1034] running on GCE node
W20260808 02:08:06.941067 15509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:06.941954 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:06.942657 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:06.943845 15500 hybrid_clock.cc:648] HybridClock initialized: now 1786154886943825 us; error 29 us; skew 500 ppm
I20260808 02:08:06.945932 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.941929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.5.209.124:33253)
0808 02:08:06.942085 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:06.942090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:06.942217 (+ 127us) server_negotiation.cc:407] Connection header received
0808 02:08:06.942395 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:06.942398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:06.942465 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:06.942567 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:06.942821 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.943390 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.944553 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.945266 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.945492 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.945675 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:06.945716 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:08:06.945784 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:08:06.945955 15512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:06.941842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56208)
0808 02:08:06.942113 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:06.942128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:06.942281 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:06.942582 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:06.942585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:06.942604 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:06.942714 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.942720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.943510 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.943514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.944428 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:06.944439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:06.945480 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:06.945484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:06.945590 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:06.945616 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:08:06.945689 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:08:06.945795 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260808 02:08:06.946771 15069 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Aug 08 02:08:06 dist-test-slave-bw88 krb5kdc[13916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:06.947497 15500 init.cc:377] Logged in from keytab as oryx/127.5.209.67@KRBTEST.COM (short username oryx)
I20260808 02:08:06.947574 15514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795ba3cf597c4d358efae359dfcf13b6. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:06.947733 15514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:08:06.948468 15500 webserver.cc:460] Webserver started at http://127.5.209.67:34289/ using document root <none> and password file <none>
I20260808 02:08:06.949054 15500 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:06.949177 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:06.953552 15500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:08:06.955631 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:06.956880 15500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:06.957015 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "ef0e65ae24e44b8d9f5daff7e734767b"
format_stamp: "Formatted at 2026-08-08 02:08:05 on dist-test-slave-bw88"
server_key: "e144457152b9fde96b821b9572991349"
server_key_iv: "8f3c77b7a93c676827dc54cc31b8f27e"
server_key_version: "encryptionkey@0"
I20260808 02:08:06.957625 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:06.997217 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:07.000713 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:07.000926 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:07.001682 15500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:07.002717 15500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:07.002766 15500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:07.002837 15500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:07.002869 15500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:07.012878 15500 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:43931
I20260808 02:08:07.012997 15681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:43931 every 8 connection(s)
I20260808 02:08:07.014092 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:07.016150 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 15500
I20260808 02:08:07.029887 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.5.209.126:35645)
0808 02:08:07.019234 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.019239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.019254 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:07.019305 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.019307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.019373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.019470 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.020573 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.021510 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.023000 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.023338 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.023429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.025268 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.025274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.025279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.025317 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.027675 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.028136 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.028143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.028147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.028237 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.028561 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.028564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.028566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.028773 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.028897 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.029526 (+ 629us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.029669 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":70,"threads_started":1}
Aug 08 02:08:07 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.124@KRBTEST.COM
Aug 08 02:08:07 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:07.033239 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.5.209.124:33253)
0808 02:08:07.020870 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.020875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.020892 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:08:07.020936 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.020942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.020999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.021083 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.022035 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.022878 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.024279 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.024633 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.024731 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.026779 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.026782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.026785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.026811 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.029425 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.031233 (+ 1808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.031237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.031239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.031329 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.031652 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.031657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.031659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.031929 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.032111 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.032852 (+ 741us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.032991 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":1822,"thread_start_us":67,"threads_started":1}
I20260808 02:08:07.033311 13977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:08:07.037204 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:54363 (local address 127.5.209.124:33253)
0808 02:08:07.019113 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.019118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.024094 (+ 4976us) server_negotiation.cc:407] Connection header received
0808 02:08:07.024275 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.024278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.024324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.024370 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.024961 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.025780 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.027097 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.027458 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.027561 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.032710 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.032714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.032717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.032746 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.034472 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.035206 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.035209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.035211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.035313 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.035829 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.035833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.035834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.036071 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.036147 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.036908 (+ 761us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.037027 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 02:08:07.037520 13977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:07.038295 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:36209 (local address 127.5.209.126:35645)
0808 02:08:07.018552 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.018557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.019782 (+ 1225us) server_negotiation.cc:407] Connection header received
0808 02:08:07.020341 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.020344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.020402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.020461 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.022025 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.022791 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.027635 (+ 4844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.027973 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.028064 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.034668 (+ 6604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.034672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.034675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.034701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.036433 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.036945 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.036948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.036950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.037033 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.037404 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.037408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.037409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.037616 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.037700 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.038040 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.038165 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260808 02:08:07.038849 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.5.209.67:54363)
0808 02:08:07.024003 (+ 5414us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.024020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.024159 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.024490 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.024493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.024707 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:07.024835 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.024843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.025943 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.025946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.026962 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.026973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.027782 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.027788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.028020 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.028641 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:07.028664 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:07.032537 (+ 3873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:07.034693 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.034700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.034716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.035081 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.035462 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.035467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.035471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.035698 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.036243 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:07.036249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:07.036597 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.037429 (+ 832us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.037792 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5322,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260808 02:08:07.038849 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:36209)
0808 02:08:07.019698 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.019712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.020209 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.020653 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.020657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.021755 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:07.021888 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.021896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.022974 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.022977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.026379 (+ 3402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.026391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.028223 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.028227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.028343 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.029127 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:07.029147 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:07.032297 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:07.036581 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.036584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.036586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.036838 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.037161 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.037164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.037165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.037254 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.037763 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:07.037766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:07.037850 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.038421 (+ 571us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.038558 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"thread_start_us":143,"threads_started":1}
I20260808 02:08:07.040009 15686 heartbeater.cc:347] Connected to a master server at 127.5.209.124:33253
I20260808 02:08:07.040223 15687 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35645
I20260808 02:08:07.040289 15686 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:07.040315 15687 heartbeater.cc:464] Registering TS with master...
Aug 08 02:08:07 dist-test-slave-bw88 krb5kdc[13916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154886, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.125@KRBTEST.COM
I20260808 02:08:07.040916 15687 heartbeater.cc:511] Master 127.5.209.126:35645 requested a full tablet report, sending...
I20260808 02:08:07.040916 15686 heartbeater.cc:511] Master 127.5.209.124:33253 requested a full tablet report, sending...
I20260808 02:08:07.042104 15059 ts_manager.cc:194] Registered new tserver with Master: ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:07.042207 13976 ts_manager.cc:194] Re-registered known tserver with Master: ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:07.042948 13976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:36209
I20260808 02:08:07.045482 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.018971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.5.209.125:38817)
0808 02:08:07.019627 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.019632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.019645 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:07.019682 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.019685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.019751 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.019967 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.021041 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.021987 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.023203 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.023568 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.023661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.029975 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.029981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.029988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.030022 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.042817 (+ 12795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.043409 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.043413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.043414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.043517 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.043840 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.043843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.043845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.044078 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.045070 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.045149 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.045251 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"spinlock_wait_cycles":5760,"thread_start_us":104,"threads_started":1}
I20260808 02:08:07.045650 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.016937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.5.209.67:54975)
0808 02:08:07.017532 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.017584 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.019312 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.020171 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.020180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.020765 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:07.021437 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.021456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.026537 (+ 5081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.026543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.027474 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.027485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.029817 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.029821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.029918 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.030915 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:07.030930 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:07.040747 (+ 9817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:07.044133 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.044136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.044138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.044366 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.044662 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:07.044665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:07.044667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:07.044744 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:07.045143 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:07.045145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:07.045225 (+ 80us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.045307 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.045467 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260808 02:08:07.045737 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.016133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:54975 (local address 127.5.209.125:38817)
0808 02:08:07.016327 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.016333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.017719 (+ 1386us) server_negotiation.cc:407] Connection header received
0808 02:08:07.019556 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.019561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.019634 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.019696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.023295 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.024565 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.028568 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.028907 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.028981 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.040934 (+ 11953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.040938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.040940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.040968 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.044002 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.044468 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.044472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.044474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.044551 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.044839 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.044842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.044844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.045022 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.045082 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.045453 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.045600 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":3456}
I20260808 02:08:07.046506 15688 heartbeater.cc:347] Connected to a master server at 127.5.209.125:38817
I20260808 02:08:07.046617 15688 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:07.046834 15688 heartbeater.cc:511] Master 127.5.209.125:38817 requested a full tablet report, sending...
I20260808 02:08:07.047639 14810 ts_manager.cc:194] Registered new tserver with Master: ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:07.047590 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: NON_VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: true } } }
I20260808 02:08:07.047726 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:07.051222 15711 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: attempting to promote NON_VOTER 19f3be198c5247c994d327ca678ae2f4 to VOTER
I20260808 02:08:07.051584 15711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } }
I20260808 02:08:07.054646 14824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:08:07.055028 14759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:08:07.055290 15069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 LEARNER]: Refusing update from remote peer 795ba3cf597c4d358efae359dfcf13b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 02:08:07.055689 14759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 02:08:07.056936 15712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.057387 14824 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.058529 15069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.059252 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.059459 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:07.060740 15514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.060854 15514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:07.062253 15712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.062388 15712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:07.062439 15734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.062593 15734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:07.062700 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "795ba3cf597c4d358efae359dfcf13b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:07.062826 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:07.067355 15512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.060555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:40919 (local address 127.0.0.1:46062)
0808 02:08:07.060829 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.060845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.060926 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.063148 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.063152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.063172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:07.063270 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.063276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.064600 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.064603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.065785 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.065795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.066917 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.066922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.067018 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.067057 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.067129 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.067221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":22,"thread_start_us":123,"threads_started":1}
I20260808 02:08:07.067564 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.061269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.5.209.65:40919)
0808 02:08:07.061587 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.061595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.061621 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:08:07.061722 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.061730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.061944 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.063198 (+ 1254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:07.063370 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.064481 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.065923 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.066761 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.066979 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.067246 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.067302 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.067410 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":141,"threads_started":1}
I20260808 02:08:07.067893 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.061740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:39221 (local address 127.0.0.1:54896)
0808 02:08:07.062082 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.062098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.062212 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.063545 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.063548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.063567 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:07.063658 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.063665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.064949 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.064951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.066012 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.066022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.066836 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.066839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.067611 (+ 772us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.067647 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.067713 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.067777 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260808 02:08:07.068365 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.061911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.5.209.66:39221)
0808 02:08:07.063015 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.063058 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.063084 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:08:07.063166 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.063172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.063381 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.063545 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:07.063803 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.064834 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.066141 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.066711 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.066835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.067005 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.068092 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.068208 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":103,"threads_started":1}
I20260808 02:08:07.068538 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.060304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:43931 (local address 127.0.0.1:42236)
0808 02:08:07.060829 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.061598 (+ 769us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.062231 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.064206 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.064209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.064229 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:07.064318 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.064327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.065738 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.065741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.066909 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.066918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.068152 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.068156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.068244 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.068268 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.068321 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.068390 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"spinlock_wait_cycles":896}
I20260808 02:08:07.069921 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.060583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.5.209.67:43931)
0808 02:08:07.063798 (+ 3215us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.063812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.063829 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:08:07.063885 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.063888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.064027 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.064228 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:07.064413 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.065609 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.067063 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.068014 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.069572 (+ 1558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.069672 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.069722 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.069780 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3133,"thread_start_us":113,"threads_started":1}
I20260808 02:08:07.070717 15210 tablet_service.cc:1467] Processing CreateTablet for tablet e89f61387c5d4da8ad35cf74c742df1e (DEFAULT_TABLE table=test-table [id=b031815e5a1b4d9185e438e060b1b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:07.071358 15400 tablet_service.cc:1467] Processing CreateTablet for tablet e89f61387c5d4da8ad35cf74c742df1e (DEFAULT_TABLE table=test-table [id=b031815e5a1b4d9185e438e060b1b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:07.072016 15210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f61387c5d4da8ad35cf74c742df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:07.072467 15400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f61387c5d4da8ad35cf74c742df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:07.072459 15589 tablet_service.cc:1467] Processing CreateTablet for tablet e89f61387c5d4da8ad35cf74c742df1e (DEFAULT_TABLE table=test-table [id=b031815e5a1b4d9185e438e060b1b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:07.073448 15589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f61387c5d4da8ad35cf74c742df1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:07.080618 15741 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Bootstrap starting.
I20260808 02:08:07.081794 15742 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Bootstrap starting.
I20260808 02:08:07.083071 15743 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Bootstrap starting.
I20260808 02:08:07.084657 15741 tablet_bootstrap.cc:654] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:07.084657 15742 tablet_bootstrap.cc:654] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:07.085417 15743 tablet_bootstrap.cc:654] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:07.086525 15741 log.cc:824] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:07.087194 15743 log.cc:824] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:07.087172 15742 log.cc:824] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:07.089514 15741 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: No bootstrap required, opened a new log
I20260808 02:08:07.089726 15741 ts_tablet_manager.cc:1397] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 02:08:07.090377 15743 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: No bootstrap required, opened a new log
I20260808 02:08:07.090567 15743 ts_tablet_manager.cc:1397] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 02:08:07.090831 15742 tablet_bootstrap.cc:492] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: No bootstrap required, opened a new log
I20260808 02:08:07.091068 15742 ts_tablet_manager.cc:1397] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260808 02:08:07.097376 15741 raft_consensus.cc:348] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.097445 15743 raft_consensus.cc:348] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.097713 15741 raft_consensus.cc:374] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:07.097712 15743 raft_consensus.cc:374] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:07.097781 15743 raft_consensus.cc:729] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5d1898dce1416ca5420bfb462323ac, State: Initialized, Role: FOLLOWER
I20260808 02:08:07.097781 15741 raft_consensus.cc:729] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0e65ae24e44b8d9f5daff7e734767b, State: Initialized, Role: FOLLOWER
I20260808 02:08:07.098289 15743 consensus_queue.cc:260] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.098289 15741 consensus_queue.cc:260] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.099460 15741 ts_tablet_manager.cc:1428] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260808 02:08:07.099464 15743 ts_tablet_manager.cc:1428] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260808 02:08:07.099671 15308 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
I20260808 02:08:07.100739 15742 raft_consensus.cc:348] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.101053 15742 raft_consensus.cc:374] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:07.101164 15742 raft_consensus.cc:729] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27de2be8517e4430b6535b92e7d725c4, State: Initialized, Role: FOLLOWER
I20260808 02:08:07.101717 15742 consensus_queue.cc:260] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.102258 15687 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
I20260808 02:08:07.103107 15742 ts_tablet_manager.cc:1428] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260808 02:08:07.104984 15493 heartbeater.cc:503] Master 127.5.209.126:35645 was elected leader, sending a full tablet report...
I20260808 02:08:07.125397 15748 raft_consensus.cc:482] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:07.125748 15748 raft_consensus.cc:504] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.127806 15748 leader_election.cc:288] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919), 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:07.133749 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.128821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:39871 (local address 127.5.209.65:40919)
0808 02:08:07.128955 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.128959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.128975 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:07.129024 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.129026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.129063 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.129156 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:07.129671 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.130371 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.132415 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.133028 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.133161 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.133217 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.133528 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.133606 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:08:07.133772 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.128227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:36475 (local address 127.5.209.66:39221)
0808 02:08:07.128476 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.128483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.128498 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:07.128553 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.128564 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.128623 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.128700 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:07.129176 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.130129 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.132415 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.133028 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.133161 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.133217 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.133530 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.133618 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260808 02:08:07.133836 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.128072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:39221 (local address 127.5.209.67:36475)
0808 02:08:07.128330 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.128345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.128438 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.128738 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.128742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.128765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:07.128998 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.129007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.130261 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.130265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.132271 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.132279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.133159 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.133162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.133399 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.133425 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.133506 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.133606 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":27}
I20260808 02:08:07.133880 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.128260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:40919 (local address 127.5.209.67:39871)
0808 02:08:07.128575 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:07.128586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:07.128640 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:07.129269 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:07.129272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:07.129310 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:07.129513 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.129522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.130491 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.130495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.132267 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:07.132279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.133159 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.133162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:07.133395 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.133431 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:08:07.133509 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:08:07.133606 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":74}
I20260808 02:08:07.134626 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f61387c5d4da8ad35cf74c742df1e" candidate_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27de2be8517e4430b6535b92e7d725c4" is_pre_election: true
I20260808 02:08:07.134781 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f61387c5d4da8ad35cf74c742df1e" candidate_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5d1898dce1416ca5420bfb462323ac" is_pre_election: true
I20260808 02:08:07.134980 15426 raft_consensus.cc:2454] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0e65ae24e44b8d9f5daff7e734767b in term 0.
I20260808 02:08:07.135020 15234 raft_consensus.cc:2454] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0e65ae24e44b8d9f5daff7e734767b in term 0.
I20260808 02:08:07.135492 15524 leader_election.cc:302] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27de2be8517e4430b6535b92e7d725c4, ef0e65ae24e44b8d9f5daff7e734767b; no voters:
I20260808 02:08:07.135769 15748 raft_consensus.cc:2781] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:07.135869 15748 raft_consensus.cc:482] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:07.135926 15748 raft_consensus.cc:3037] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:07.137320 15748 raft_consensus.cc:504] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.137804 15748 leader_election.cc:288] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [CANDIDATE]: Term 1 election: Requested vote from peers de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919), 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221)
I20260808 02:08:07.138195 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f61387c5d4da8ad35cf74c742df1e" candidate_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5d1898dce1416ca5420bfb462323ac"
I20260808 02:08:07.138335 15234 raft_consensus.cc:3037] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:07.138332 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f61387c5d4da8ad35cf74c742df1e" candidate_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27de2be8517e4430b6535b92e7d725c4"
I20260808 02:08:07.138470 15426 raft_consensus.cc:3037] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:07.139868 15426 raft_consensus.cc:2454] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0e65ae24e44b8d9f5daff7e734767b in term 1.
I20260808 02:08:07.139915 15234 raft_consensus.cc:2454] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0e65ae24e44b8d9f5daff7e734767b in term 1.
I20260808 02:08:07.140336 15525 leader_election.cc:302] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de5d1898dce1416ca5420bfb462323ac, ef0e65ae24e44b8d9f5daff7e734767b; no voters:
I20260808 02:08:07.140599 15748 raft_consensus.cc:2781] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:07.140837 15748 raft_consensus.cc:686] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 1 LEADER]: Becoming Leader. State: Replica: ef0e65ae24e44b8d9f5daff7e734767b, State: Running, Role: LEADER
I20260808 02:08:07.141515 15748 consensus_queue.cc:237] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:07.145193 13976 catalog_manager.cc:5495] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67). New cstate: current_term: 1 leader_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:07.164839 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.160942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.5.209.67:43931)
0808 02:08:07.161093 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.161097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.161113 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:07.161341 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.161361 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.161440 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.161554 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:07.161806 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.162415 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.163302 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.163526 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.163660 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.163990 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:07.164422 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:07.164555 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.164633 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.164691 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:08:07.171614 15426 raft_consensus.cc:1261] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Refusing update from remote peer ef0e65ae24e44b8d9f5daff7e734767b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:07.171777 15234 raft_consensus.cc:1261] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Refusing update from remote peer ef0e65ae24e44b8d9f5daff7e734767b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:07.172408 15751 consensus_queue.cc:1035] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:07.172605 15748 consensus_queue.cc:1035] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:07.181166 15756 mvcc.cc:205] Tried to move back new op lower bound from 7316090417846681600 to 7316090417734352896. Current Snapshot: MvccSnapshot[applied={T|T < 7316090417846681600}]
I20260808 02:08:07.187233 15757 mvcc.cc:205] Tried to move back new op lower bound from 7316090417846681600 to 7316090417734352896. Current Snapshot: MvccSnapshot[applied={T|T < 7316090417846681600}]
I20260808 02:08:07.193918 13977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:07.194082 13977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:07.199468 13977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 795ba3cf597c4d358efae359dfcf13b6: Sending DeleteTablet for 3 replicas of tablet e89f61387c5d4da8ad35cf74c742df1e
I20260808 02:08:07.200265 15589 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f61387c5d4da8ad35cf74c742df1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:07 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:42236
I20260808 02:08:07.200374 15210 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f61387c5d4da8ad35cf74c742df1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:07 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:46062
I20260808 02:08:07.201051 15771 tablet_replica.cc:330] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: stopping tablet replica
I20260808 02:08:07.201014 15768 tablet_replica.cc:330] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: stopping tablet replica
I20260808 02:08:07.201169 15400 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f61387c5d4da8ad35cf74c742df1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:07 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:54896
I20260808 02:08:07.201555 15771 raft_consensus.cc:2229] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:07.201745 15768 raft_consensus.cc:2229] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:07.201900 15771 raft_consensus.cc:2258] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:07.202773 15768 raft_consensus.cc:2258] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:07.204325 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 13933
I20260808 02:08:07.212679 15768 ts_tablet_manager.cc:1905] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:07.215683 15771 ts_tablet_manager.cc:1905] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:07.218765 15771 ts_tablet_manager.cc:1918] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:07.218870 15771 log.cc:1196] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/e89f61387c5d4da8ad35cf74c742df1e
I20260808 02:08:07.219223 15771 ts_tablet_manager.cc:1939] T e89f61387c5d4da8ad35cf74c742df1e P de5d1898dce1416ca5420bfb462323ac: Deleting consensus metadata
I20260808 02:08:07.220299 15768 ts_tablet_manager.cc:1918] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:07.220425 15768 log.cc:1196] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/e89f61387c5d4da8ad35cf74c742df1e
I20260808 02:08:07.221778 15768 ts_tablet_manager.cc:1939] T e89f61387c5d4da8ad35cf74c742df1e P ef0e65ae24e44b8d9f5daff7e734767b: Deleting consensus metadata
W20260808 02:08:07.224383 15524 connection.cc:535] server connection from 127.0.0.1:42236 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 02:08:07.224436 15151 connection.cc:535] server connection from 127.0.0.1:46062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:08:07.224905 15772 tablet_replica.cc:330] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: stopping tablet replica
I20260808 02:08:07.225278 15772 raft_consensus.cc:2229] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:07.225685 15772 raft_consensus.cc:2258] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:07.226109 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.204115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.5.209.125:38817)
0808 02:08:07.204237 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.204242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.204257 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:07.204303 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.204306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.204361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.204467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.215072 (+ 10605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.216020 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.217697 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.218120 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.218216 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.220921 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.220927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.220933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.220967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.223141 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.223828 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.223835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.223839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.223951 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.224853 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.224856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.224858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.225108 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.225219 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.225768 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.225903 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:08:07.230386 15772 ts_tablet_manager.cc:1905] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:07.233651 15772 ts_tablet_manager.cc:1918] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:07.233762 15772 log.cc:1196] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/e89f61387c5d4da8ad35cf74c742df1e
I20260808 02:08:07.234187 15772 ts_tablet_manager.cc:1939] T e89f61387c5d4da8ad35cf74c742df1e P 27de2be8517e4430b6535b92e7d725c4: Deleting consensus metadata
W20260808 02:08:07.235548 15343 connection.cc:441] server connection from 127.0.0.1:54896 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54896 (request call id 1) could send its response
I20260808 02:08:07.237860 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.203740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.5.209.124:33253)
0808 02:08:07.203875 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.203882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.227998 (+ 24116us) server_negotiation.cc:407] Connection header received
0808 02:08:07.228217 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.228221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.228290 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.228344 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.229198 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.229981 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.231226 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.231552 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.231637 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.233535 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.233542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.233545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.233577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.235802 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.236265 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.236269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.236271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.236367 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.236719 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.236723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.236724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.236961 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.237086 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.237517 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.237663 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:08:07.240914 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.229710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.5.209.125:38817)
0808 02:08:07.229810 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.229814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.230909 (+ 1095us) server_negotiation.cc:407] Connection header received
0808 02:08:07.231078 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.231081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.231124 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.231172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.232646 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.233512 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.234663 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.235005 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.235086 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.236722 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.236725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.236728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.236754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.238691 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.239481 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.239485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.239487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.239585 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.239916 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.239920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.239922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.240158 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.240264 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.240636 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.240756 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:08:07.246028 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:07.230074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.5.209.124:33253)
0808 02:08:07.230194 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:07.230198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:07.236664 (+ 6466us) server_negotiation.cc:407] Connection header received
0808 02:08:07.236832 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:07.236836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:07.236879 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:07.236928 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:07.239124 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.239785 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.240627 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:07.240872 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:07.241013 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:07.242786 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:07.242789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:07.242791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:07.242812 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:07.244310 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.244787 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.244790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.244791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.244866 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:07.245148 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:07.245151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:07.245152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:07.245310 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:07.245486 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:07.245739 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 02:08:07.245861 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:08:08.190694 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.190261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:37857)
0808 02:08:08.190506 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.190593 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20260808 02:08:08.191022 15308 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.192034 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.191642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:49541)
0808 02:08:08.191880 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.191954 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
W20260808 02:08:08.192360 15493 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.192881 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.192502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:41599)
0808 02:08:08.192734 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.192804 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
W20260808 02:08:08.193154 15687 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:35645 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.698966 15789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795ba3cf597c4d358efae359dfcf13b6)
I20260808 02:08:08.699333 15789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } }
I20260808 02:08:08.699427 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795ba3cf597c4d358efae359dfcf13b6)
I20260808 02:08:08.699581 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } }
I20260808 02:08:08.700577 15790 leader_election.cc:288] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645), 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124:33253)
I20260808 02:08:08.701004 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.700268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36012)
0808 02:08:08.700802 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.700898 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":81,"threads_started":1}
W20260808 02:08:08.701234 14798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.701412 15789 leader_election.cc:288] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645), 3b0dd0822d5d44918988f8e918ba3898 (127.5.209.125:38817)
I20260808 02:08:08.701478 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.700734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36028)
0808 02:08:08.701223 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.701302 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":77,"threads_started":1}
W20260808 02:08:08.701797 15040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.702531 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.702256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36040)
0808 02:08:08.702395 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.702458 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 02:08:08.702687 14798 leader_election.cc:334] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645): Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.702777 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.702517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36048)
0808 02:08:08.702665 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.702717 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260808 02:08:08.702870 15040 leader_election.cc:334] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645): Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.705283 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.700805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:33253 (local address 127.0.0.1:56256)
0808 02:08:08.701687 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.701704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.701826 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.702216 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.702218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.702233 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.702313 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.702324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.703353 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.703355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.704169 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.704178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.704868 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.704956 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.704978 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.705056 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.705138 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":68,"threads_started":1}
I20260808 02:08:08.705287 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.700910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.5.209.124:33253)
0808 02:08:08.701248 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.701252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.701768 (+ 516us) server_negotiation.cc:407] Connection header received
0808 02:08:08.701973 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.701979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.702030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.702111 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.702439 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.703229 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704302 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.704739 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704859 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.704924 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.705088 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.705154 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":72,"threads_started":1}
I20260808 02:08:08.705497 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.701231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:38817 (local address 127.0.0.1:52878)
0808 02:08:08.701627 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.701641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.701740 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.702163 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.702166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.702184 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.702274 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.702280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.703376 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.703378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.704168 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.704176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.704994 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.705100 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.705125 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.705199 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.705267 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":55,"threads_started":1}
I20260808 02:08:08.705752 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b0dd0822d5d44918988f8e918ba3898" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19f3be198c5247c994d327ca678ae2f4" is_pre_election: true
I20260808 02:08:08.705937 15069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0dd0822d5d44918988f8e918ba3898 in term 1.
I20260808 02:08:08.706239 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.701411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.5.209.125:38817)
0808 02:08:08.701744 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.701795 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.701810 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:08.701865 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.701867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.701944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.702025 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.702397 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.703229 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704298 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.704758 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.704876 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.704933 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.705231 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.705315 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":97,"threads_started":1}
I20260808 02:08:08.706290 14796 leader_election.cc:302] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19f3be198c5247c994d327ca678ae2f4, 3b0dd0822d5d44918988f8e918ba3898; no voters: 795ba3cf597c4d358efae359dfcf13b6
I20260808 02:08:08.706462 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:08:08.706525 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795ba3cf597c4d358efae359dfcf13b6)
I20260808 02:08:08.706588 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:08:08.706660 14824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19f3be198c5247c994d327ca678ae2f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b0dd0822d5d44918988f8e918ba3898" is_pre_election: true
I20260808 02:08:08.707684 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } }
I20260808 02:08:08.707966 14824 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 19f3be198c5247c994d327ca678ae2f4 in current term 2: Already voted for candidate 3b0dd0822d5d44918988f8e918ba3898 in this term.
I20260808 02:08:08.708076 15790 leader_election.cc:288] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 election: Requested vote from peers 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645), 19f3be198c5247c994d327ca678ae2f4 (127.5.209.124:33253)
I20260808 02:08:08.708410 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.708170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36054)
0808 02:08:08.708280 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.708331 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:08:08.708462 15069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b0dd0822d5d44918988f8e918ba3898" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19f3be198c5247c994d327ca678ae2f4"
I20260808 02:08:08.708581 15069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:08:08.708741 15035 leader_election.cc:302] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19f3be198c5247c994d327ca678ae2f4; no voters: 3b0dd0822d5d44918988f8e918ba3898, 795ba3cf597c4d358efae359dfcf13b6
I20260808 02:08:08.709754 15069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0dd0822d5d44918988f8e918ba3898 in term 2.
I20260808 02:08:08.709946 15789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:08:08.710008 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.709775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36056)
0808 02:08:08.709887 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.709945 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:08:08.710072 14796 leader_election.cc:302] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f3be198c5247c994d327ca678ae2f4, 3b0dd0822d5d44918988f8e918ba3898; no voters:
I20260808 02:08:08.710258 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 2 FOLLOWER]: Leader election won for term 2
W20260808 02:08:08.710289 14798 leader_election.cc:334] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645): Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
I20260808 02:08:08.710496 15790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [term 2 LEADER]: Becoming Leader. State: Replica: 3b0dd0822d5d44918988f8e918ba3898, State: Running, Role: LEADER
I20260808 02:08:08.710783 15790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } }
I20260808 02:08:08.711753 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b0dd0822d5d44918988f8e918ba3898. Latest consensus state: current_term: 2 leader_uuid: "3b0dd0822d5d44918988f8e918ba3898" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:08.711874 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:08.712179 15803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:08.714484 15803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:08.714879 15803 catalog_manager.cc:1259] Loaded cluster ID: 572990ab8fe046a59afc1da1d173d8ce
I20260808 02:08:08.714923 15803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:08.715468 15803 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:08.715865 15803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Loaded TSK: 0
I20260808 02:08:08.716534 15803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:08:08.763965 14810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:08:08.767335 14810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:08.770458 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.770115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36076)
0808 02:08:08.770236 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.770294 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 02:08:08.770484 15069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [term 2 FOLLOWER]: Refusing update from remote peer 3b0dd0822d5d44918988f8e918ba3898: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 02:08:08.771129 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.770909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.0.0.1:36084)
0808 02:08:08.771021 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.771077 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 02:08:08.771036 15800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260808 02:08:08.771255 14798 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 -> Peer 795ba3cf597c4d358efae359dfcf13b6 (127.5.209.126:35645): Couldn't send request to peer 795ba3cf597c4d358efae359dfcf13b6. Status: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:08:08.772639 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b0dd0822d5d44918988f8e918ba3898. Latest consensus state: current_term: 2 leader_uuid: "3b0dd0822d5d44918988f8e918ba3898" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:08.772760 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:08.774005 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3b0dd0822d5d44918988f8e918ba3898" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:08.774143 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:08.774369 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3b0dd0822d5d44918988f8e918ba3898" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "795ba3cf597c4d358efae359dfcf13b6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35645 } } peers { permanent_uuid: "3b0dd0822d5d44918988f8e918ba3898" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 38817 } attrs { promote: false } } peers { permanent_uuid: "19f3be198c5247c994d327ca678ae2f4" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 33253 } attrs { promote: false } } }
I20260808 02:08:08.774446 15805 mvcc.cc:205] Tried to move back new op lower bound from 7316090424398495744 to 7316090424161488896. Current Snapshot: MvccSnapshot[applied={T|T < 7316090424398495744}]
I20260808 02:08:08.774490 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f3be198c5247c994d327ca678ae2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:08:08.785498 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.780731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:43931 (local address 127.0.0.1:42252)
0808 02:08:08.781032 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.781045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.781139 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.781775 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.781778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.781794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.781905 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.781912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.782961 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.782964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.784111 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.784122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.785028 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.785032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.785139 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.785177 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.785262 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.785359 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260808 02:08:08.786051 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.780067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:40919 (local address 127.0.0.1:46076)
0808 02:08:08.780221 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.780235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.780313 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.780924 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.780928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.780948 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.781072 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.781081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.782091 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.782094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.783029 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.783038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.785623 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.785627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.785746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.785769 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.785873 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.785944 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260808 02:08:08.786105 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.780233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.5.209.65:40919)
0808 02:08:08.780482 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.780504 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.780523 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:08:08.780607 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.780613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.780787 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.781199 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.781323 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.781947 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.783182 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.783692 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.783855 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.784858 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.785895 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.785982 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260808 02:08:08.786378 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.781021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.5.209.67:43931)
0808 02:08:08.781563 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.781566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.781578 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:08.781624 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.781626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.781665 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.781766 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.782035 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.782845 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.784252 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.784908 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.785011 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.785076 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.785435 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.786215 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":85,"threads_started":1}
I20260808 02:08:08.786599 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.780688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:39221 (local address 127.0.0.1:54906)
0808 02:08:08.780931 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.780944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.781009 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.782123 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.782126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.782144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.782249 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.782255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.783378 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.783382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.784698 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.784708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.785655 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.785658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.786350 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.786373 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.786445 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.786517 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":572}
I20260808 02:08:08.786990 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.780805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.5.209.66:39221)
0808 02:08:08.781323 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.781329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.781871 (+ 542us) server_negotiation.cc:407] Connection header received
0808 02:08:08.781927 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.781931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.781991 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.782360 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.782531 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.783261 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.784838 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.785508 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.785748 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.785815 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.786798 (+ 983us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.786891 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":70,"threads_started":1}
I20260808 02:08:08.786940 15589 tablet_service.cc:1467] Processing CreateTablet for tablet fb12a79fda2f452e9a4229e3cff93722 (DEFAULT_TABLE table=test-table [id=0c5edef3e30d4ca0b22cd328f0b7176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:08.787494 15589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12a79fda2f452e9a4229e3cff93722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:08.787650 15400 tablet_service.cc:1467] Processing CreateTablet for tablet fb12a79fda2f452e9a4229e3cff93722 (DEFAULT_TABLE table=test-table [id=0c5edef3e30d4ca0b22cd328f0b7176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:08.787922 15400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12a79fda2f452e9a4229e3cff93722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:08.790390 15819 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Bootstrap starting.
I20260808 02:08:08.790526 15821 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Bootstrap starting.
I20260808 02:08:08.790733 15210 tablet_service.cc:1467] Processing CreateTablet for tablet fb12a79fda2f452e9a4229e3cff93722 (DEFAULT_TABLE table=test-table [id=0c5edef3e30d4ca0b22cd328f0b7176a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:08.791041 15210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12a79fda2f452e9a4229e3cff93722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:08.791464 15819 tablet_bootstrap.cc:654] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:08.791486 15821 tablet_bootstrap.cc:654] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:08.792994 15821 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: No bootstrap required, opened a new log
I20260808 02:08:08.793089 15821 ts_tablet_manager.cc:1397] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:08.793614 15821 raft_consensus.cc:348] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.793744 15821 raft_consensus.cc:374] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:08.793797 15821 raft_consensus.cc:729] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27de2be8517e4430b6535b92e7d725c4, State: Initialized, Role: FOLLOWER
I20260808 02:08:08.793932 15826 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac: Bootstrap starting.
I20260808 02:08:08.793943 15821 consensus_queue.cc:260] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.794299 15821 ts_tablet_manager.cc:1428] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 02:08:08.795207 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.794769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:46309)
0808 02:08:08.795028 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.795098 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":126,"threads_started":1}
I20260808 02:08:08.795377 15819 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: No bootstrap required, opened a new log
I20260808 02:08:08.795467 15819 ts_tablet_manager.cc:1397] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 02:08:08.795930 15819 raft_consensus.cc:348] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.796046 15819 raft_consensus.cc:374] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:08.796085 15819 raft_consensus.cc:729] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0e65ae24e44b8d9f5daff7e734767b, State: Initialized, Role: FOLLOWER
I20260808 02:08:08.796216 15819 consensus_queue.cc:260] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.796507 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.796025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:38971)
0808 02:08:08.796383 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.796441 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260808 02:08:08.796527 15819 ts_tablet_manager.cc:1428] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:08.797425 15495 heartbeater.cc:503] Master 127.5.209.125:38817 was elected leader, sending a full tablet report...
I20260808 02:08:08.797871 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.797290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:47593)
0808 02:08:08.797710 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.797772 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":129,"threads_started":1}
I20260808 02:08:08.798379 15688 heartbeater.cc:503] Master 127.5.209.125:38817 was elected leader, sending a full tablet report...
I20260808 02:08:08.798683 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.798265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:44079)
0808 02:08:08.798535 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.798596 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260808 02:08:08.799626 15826 tablet_bootstrap.cc:654] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:08.800921 15826 tablet_bootstrap.cc:492] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac: No bootstrap required, opened a new log
I20260808 02:08:08.800998 15826 ts_tablet_manager.cc:1397] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 02:08:08.801440 15826 raft_consensus.cc:348] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.801533 15826 raft_consensus.cc:374] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:08.801555 15826 raft_consensus.cc:729] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5d1898dce1416ca5420bfb462323ac, State: Initialized, Role: FOLLOWER
I20260808 02:08:08.801687 15826 consensus_queue.cc:260] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.801951 15826 ts_tablet_manager.cc:1428] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:08.802802 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.802340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:35215)
0808 02:08:08.802625 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.802689 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260808 02:08:08.803509 15309 heartbeater.cc:503] Master 127.5.209.125:38817 was elected leader, sending a full tablet report...
I20260808 02:08:08.803665 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.803223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:41083)
0808 02:08:08.803484 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.803542 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260808 02:08:08.812508 15827 raft_consensus.cc:482] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:08.812611 15827 raft_consensus.cc:504] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.813859 15827 leader_election.cc:288] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919), ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:08.817665 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.814222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:40927 (local address 127.5.209.67:43931)
0808 02:08:08.814340 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.814345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.814473 (+ 128us) server_negotiation.cc:407] Connection header received
0808 02:08:08.814536 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.814542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.814593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.814659 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.815014 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.815548 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.816611 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.817017 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.817118 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.817197 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.817453 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.817534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:08:08.817689 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.814086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:57681 (local address 127.5.209.65:40919)
0808 02:08:08.814203 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.814210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.814327 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:08:08.814439 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.814445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.814501 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.814589 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:08.814873 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.815429 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.816667 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.817088 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.817209 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.817261 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.817497 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.817568 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:08:08.817698 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.814029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:40919 (local address 127.5.209.66:57681)
0808 02:08:08.814202 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.814216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.814321 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.814613 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.814616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.814651 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.814761 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.814766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.815549 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.815553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.816490 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.816500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.817228 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.817236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.817321 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.817382 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.817466 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.817537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":28}
I20260808 02:08:08.817745 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.814061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:43931 (local address 127.5.209.66:40927)
0808 02:08:08.814357 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.814372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:08.814462 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:08.814735 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:08.814741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:08.814760 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:08.814892 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.814899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.815661 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.815663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.816484 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:08.816493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.817155 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.817162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:08.817267 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.817297 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:08:08.817428 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 02:08:08.817524 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260808 02:08:08.818140 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12a79fda2f452e9a4229e3cff93722" candidate_uuid: "27de2be8517e4430b6535b92e7d725c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5d1898dce1416ca5420bfb462323ac" is_pre_election: true
I20260808 02:08:08.818295 15234 raft_consensus.cc:2454] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27de2be8517e4430b6535b92e7d725c4 in term 0.
I20260808 02:08:08.818490 15614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12a79fda2f452e9a4229e3cff93722" candidate_uuid: "27de2be8517e4430b6535b92e7d725c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" is_pre_election: true
I20260808 02:08:08.818686 15343 leader_election.cc:302] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27de2be8517e4430b6535b92e7d725c4, de5d1898dce1416ca5420bfb462323ac; no voters:
I20260808 02:08:08.818747 15614 raft_consensus.cc:2454] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27de2be8517e4430b6535b92e7d725c4 in term 0.
I20260808 02:08:08.818984 15827 raft_consensus.cc:2781] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:08.819078 15827 raft_consensus.cc:482] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:08.819123 15827 raft_consensus.cc:3037] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:08.820394 15827 raft_consensus.cc:504] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.820736 15827 leader_election.cc:288] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [CANDIDATE]: Term 1 election: Requested vote from peers de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919), ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931)
I20260808 02:08:08.821115 15234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12a79fda2f452e9a4229e3cff93722" candidate_uuid: "27de2be8517e4430b6535b92e7d725c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de5d1898dce1416ca5420bfb462323ac"
I20260808 02:08:08.821247 15234 raft_consensus.cc:3037] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:08.821465 15614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12a79fda2f452e9a4229e3cff93722" candidate_uuid: "27de2be8517e4430b6535b92e7d725c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0e65ae24e44b8d9f5daff7e734767b"
I20260808 02:08:08.821591 15614 raft_consensus.cc:3037] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:08.822758 15614 raft_consensus.cc:2454] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27de2be8517e4430b6535b92e7d725c4 in term 1.
I20260808 02:08:08.823060 15341 leader_election.cc:302] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27de2be8517e4430b6535b92e7d725c4, ef0e65ae24e44b8d9f5daff7e734767b; no voters:
I20260808 02:08:08.823240 15827 raft_consensus.cc:2781] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:08.823441 15827 raft_consensus.cc:686] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 1 LEADER]: Becoming Leader. State: Replica: 27de2be8517e4430b6535b92e7d725c4, State: Running, Role: LEADER
I20260808 02:08:08.824074 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.823872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:56193)
0808 02:08:08.823966 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.824015 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 02:08:08.824100 15234 raft_consensus.cc:2454] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27de2be8517e4430b6535b92e7d725c4 in term 1.
I20260808 02:08:08.824139 15827 consensus_queue.cc:237] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } }
I20260808 02:08:08.826508 14810 catalog_manager.cc:5495] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "27de2be8517e4430b6535b92e7d725c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27de2be8517e4430b6535b92e7d725c4" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39221 } health_report { overall_health: HEALTHY } } }
I20260808 02:08:08.851861 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.847991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.5.209.66:39221)
0808 02:08:08.848140 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:08.848144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:08.848160 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:08.848242 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:08.848244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:08.848299 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:08.848389 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:08.848655 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.849162 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.850145 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:08.850359 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:08.850456 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:08.850901 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:08.851388 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:08.851478 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:08.851633 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:08:08.851688 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:08:08.854952 15234 raft_consensus.cc:1261] T fb12a79fda2f452e9a4229e3cff93722 P de5d1898dce1416ca5420bfb462323ac [term 1 FOLLOWER]: Refusing update from remote peer 27de2be8517e4430b6535b92e7d725c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:08.855281 15614 raft_consensus.cc:1261] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Refusing update from remote peer 27de2be8517e4430b6535b92e7d725c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:08.855589 15843 consensus_queue.cc:1035] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de5d1898dce1416ca5420bfb462323ac" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:08.855928 15827 consensus_queue.cc:1035] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0e65ae24e44b8d9f5daff7e734767b" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:08.857832 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.857508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:44887)
0808 02:08:08.857664 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.857725 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 02:08:08.858052 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.857728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:36901)
0808 02:08:08.857886 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.857958 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 02:08:08.860704 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.859856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:57503)
0808 02:08:08.860565 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.860635 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260808 02:08:08.861836 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.861470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.65:35661)
0808 02:08:08.861713 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.861767 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 02:08:08.863572 15847 mvcc.cc:205] Tried to move back new op lower bound from 7316090424744468480 to 7316090424626069504. Current Snapshot: MvccSnapshot[applied={T|T < 7316090424744468480 or (T in {7316090424744468480})}]
I20260808 02:08:08.866165 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.865863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.66:37167)
0808 02:08:08.866027 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.866079 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:08:08.867558 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.867255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35645 (local address 127.5.209.67:37757)
0808 02:08:08.867415 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.867484 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:08:08.869477 14810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:08.869705 14810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:08.873543 14810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b0dd0822d5d44918988f8e918ba3898: Sending DeleteTablet for 3 replicas of tablet fb12a79fda2f452e9a4229e3cff93722
I20260808 02:08:08.874255 15589 tablet_service.cc:1514] Processing DeleteTablet for tablet fb12a79fda2f452e9a4229e3cff93722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:08 UTC) from {username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:42252
I20260808 02:08:08.874406 15210 tablet_service.cc:1514] Processing DeleteTablet for tablet fb12a79fda2f452e9a4229e3cff93722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:08 UTC) from {username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:46076
I20260808 02:08:08.874424 15400 tablet_service.cc:1514] Processing DeleteTablet for tablet fb12a79fda2f452e9a4229e3cff93722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:08 UTC) from {username='oryx', principal='oryx/127.5.209.125@KRBTEST.COM'} at 127.0.0.1:54906
I20260808 02:08:08.874948 15861 tablet_replica.cc:330] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: stopping tablet replica
I20260808 02:08:08.875087 15861 raft_consensus.cc:2229] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:08.875229 15861 raft_consensus.cc:2258] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:08.875423 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 15005
I20260808 02:08:08.881060 15861 ts_tablet_manager.cc:1905] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:08.885144 15861 ts_tablet_manager.cc:1918] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:08.885236 15861 log.cc:1196] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/fb12a79fda2f452e9a4229e3cff93722
I20260808 02:08:08.885593 15863 tablet_replica.cc:330] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: stopping tablet replica
I20260808 02:08:08.885617 15861 ts_tablet_manager.cc:1939] T fb12a79fda2f452e9a4229e3cff93722 P ef0e65ae24e44b8d9f5daff7e734767b: Deleting consensus metadata
I20260808 02:08:08.885735 15863 raft_consensus.cc:2229] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:08.886107 15863 raft_consensus.cc:2258] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:08.887322 14795 catalog_manager.cc:4841] TS ef0e65ae24e44b8d9f5daff7e734767b (127.5.209.67:43931): tablet fb12a79fda2f452e9a4229e3cff93722 (table test-table [id=0c5edef3e30d4ca0b22cd328f0b7176a]) successfully deleted
I20260808 02:08:08.887388 15863 ts_tablet_manager.cc:1905] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:08.889786 15863 ts_tablet_manager.cc:1918] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:08.889890 15863 log.cc:1196] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/fb12a79fda2f452e9a4229e3cff93722
I20260808 02:08:08.890169 15863 ts_tablet_manager.cc:1939] T fb12a79fda2f452e9a4229e3cff93722 P 27de2be8517e4430b6535b92e7d725c4: Deleting consensus metadata
I20260808 02:08:08.890923 14796 catalog_manager.cc:4841] TS 27de2be8517e4430b6535b92e7d725c4 (127.5.209.66:39221): tablet fb12a79fda2f452e9a4229e3cff93722 (table test-table [id=0c5edef3e30d4ca0b22cd328f0b7176a]) successfully deleted
I20260808 02:08:08.898545 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 15317
I20260808 02:08:08.898823 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:08.898501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:40919 (local address 127.0.0.1:46088)
0808 02:08:08.898670 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:08.898732 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 02:08:08.903347 14798 catalog_manager.cc:4584] TS de5d1898dce1416ca5420bfb462323ac (127.5.209.65:40919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb12a79fda2f452e9a4229e3cff93722: Network error: Client connection negotiation failed: client connection to 127.5.209.65:40919: connect: Connection refused (error 111)
I20260808 02:08:08.914508 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 15500
I20260808 02:08:08.927640 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 14625
I20260808 02:08:08.939041 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 14775
2026-08-08T02:08:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5659 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:09 dist-test-slave-bw88 krb5kdc[15868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:09 dist-test-slave-bw88 krb5kdc[15868](info): set up 2 sockets
Aug 08 02:08:09 dist-test-slave-bw88 krb5kdc[15868](info): commencing operation
krb5kdc: starting...
W20260808 02:08:10.221822 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:10 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab.
Entry for principal oryx/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab.
I20260808 02:08:10.353549 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33859
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab
--principal=oryx/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:10.471354 15887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:10.471616 15887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:10.471654 15887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:10.475699 15887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:10.475756 15887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:08:10.475773 15887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:10.475791 15887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:10.475806 15887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:10.475860 15887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:10.481812 15887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33859
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.126.keytab
--principal=oryx/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:10.482911 15887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:10.483983 15887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:10.492767 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:10.492794 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:10.492760 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:10.493157 15887 server_base.cc:1034] running on GCE node
I20260808 02:08:10.493824 15887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:10.494982 15887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:10.496151 15887 hybrid_clock.cc:648] HybridClock initialized: now 1786154890496122 us; error 41 us; skew 500 ppm
Aug 08 02:08:10 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:10.499076 15887 init.cc:377] Logged in from keytab as oryx/127.5.209.126@KRBTEST.COM (short username oryx)
I20260808 02:08:10.500150 15887 webserver.cc:460] Webserver started at http://127.5.209.126:42141/ using document root <none> and password file <none>
I20260808 02:08:10.500746 15887 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:10.500808 15887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:10.501065 15887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:10.503319 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "ce77e531976c4f4eb3c9f5167d20f91a"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "c5761ed7ff5af694b899339c17dd61a6"
server_key_iv: "0581e7e5b17d028724ccb7c54892e661"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.503929 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "ce77e531976c4f4eb3c9f5167d20f91a"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "c5761ed7ff5af694b899339c17dd61a6"
server_key_iv: "0581e7e5b17d028724ccb7c54892e661"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.507831 15887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:08:10.510437 15907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:10.511677 15887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:10.511811 15887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "ce77e531976c4f4eb3c9f5167d20f91a"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "c5761ed7ff5af694b899339c17dd61a6"
server_key_iv: "0581e7e5b17d028724ccb7c54892e661"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.511919 15887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:10.535607 15887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:10.538666 15887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:10.538892 15887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:10.547394 15887 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33859
I20260808 02:08:10.547432 15996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33859 every 8 connection(s)
I20260808 02:08:10.548413 15887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:10.551003 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 15887
I20260808 02:08:10.551159 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:10.551450 15997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:10.551587 5957 external_mini_cluster.cc:1428] Setting key ef5c34fdd570dcbe92b319b63df74b8c
I20260808 02:08:10.558568 15997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Bootstrap starting.
I20260808 02:08:10.561540 15997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Neither blocks nor log segments found. Creating new log.
Aug 08 02:08:10 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:10.562402 15997 log.cc:824] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:10.564996 15997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: No bootstrap required, opened a new log
I20260808 02:08:10.568003 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:10.553158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.5.209.126:33859)
0808 02:08:10.553643 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:10.553668 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:10.553707 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:08:10.554753 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:10.554776 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:10.555141 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:10.555460 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:10.556419 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.557532 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.558705 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.559193 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.559317 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:10.562182 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:10.562191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:10.562210 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:10.562245 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:10.564804 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.565294 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.565300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.565306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.565475 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.565777 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.565780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.565782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.566295 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:10.566469 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:10.566739 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:08:10.567044 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":202,"threads_started":1}
I20260808 02:08:10.569121 15997 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } }
I20260808 02:08:10.569398 15997 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:10.569458 15997 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce77e531976c4f4eb3c9f5167d20f91a, State: Initialized, Role: FOLLOWER
I20260808 02:08:10.569921 15997 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } }
I20260808 02:08:10.570047 15997 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:10.570140 15997 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:10.570242 15997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:10.571242 15997 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } }
I20260808 02:08:10.571596 15997 leader_election.cc:302] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce77e531976c4f4eb3c9f5167d20f91a; no voters:
I20260808 02:08:10.571974 15997 leader_election.cc:288] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:10.572122 16002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:10.572358 16002 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [term 1 LEADER]: Becoming Leader. State: Replica: ce77e531976c4f4eb3c9f5167d20f91a, State: Running, Role: LEADER
I20260808 02:08:10.572671 16002 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } }
I20260808 02:08:10.573235 15997 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:10.573853 16002 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } } }
I20260808 02:08:10.573975 16002 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:10.574210 16002 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce77e531976c4f4eb3c9f5167d20f91a. Latest consensus state: current_term: 1 leader_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce77e531976c4f4eb3c9f5167d20f91a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33859 } } }
I20260808 02:08:10.574324 16002 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:10.574637 16012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:10.578024 16012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:10.584151 16012 catalog_manager.cc:1347] Generated new cluster ID: d234218681864d12ab129774a5dbbc16
I20260808 02:08:10.584236 16012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:10.606371 16012 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:10.607736 16012 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:10.628623 16012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Generated new TSK 0
I20260808 02:08:10.629665 16012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
I20260808 02:08:10.672698 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33859
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:10.790524 16029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:10.790851 16029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:10.790933 16029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:10.796026 16029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:10.796140 16029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:10.796204 16029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:08:10.796293 16029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:10.801887 16029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:10.803056 16029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:10.804185 16029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:10.814512 16040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:10.814502 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:10.814556 16029 server_base.cc:1034] running on GCE node
W20260808 02:08:10.814502 16046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:10.815258 16029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:10.816046 16029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:10.817257 16029 hybrid_clock.cc:648] HybridClock initialized: now 1786154890817223 us; error 43 us; skew 500 ppm
Aug 08 02:08:10 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:10.820182 16029 init.cc:377] Logged in from keytab as oryx/127.5.209.65@KRBTEST.COM (short username oryx)
I20260808 02:08:10.821049 16029 webserver.cc:460] Webserver started at http://127.5.209.65:40089/ using document root <none> and password file <none>
I20260808 02:08:10.821724 16029 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:10.821789 16029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:10.822036 16029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:10.824208 16029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "cea9829756534da8a0a72f8528db57e9"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "7f53e80418343f6b092c9f97beafa9d6"
server_key_iv: "95581f20a76ec98d84f037aefdd78104"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.824795 16029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "cea9829756534da8a0a72f8528db57e9"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "7f53e80418343f6b092c9f97beafa9d6"
server_key_iv: "95581f20a76ec98d84f037aefdd78104"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.828707 16029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:10.831207 16058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:10.832445 16029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:10.832592 16029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "cea9829756534da8a0a72f8528db57e9"
format_stamp: "Formatted at 2026-08-08 02:08:10 on dist-test-slave-bw88"
server_key: "7f53e80418343f6b092c9f97beafa9d6"
server_key_iv: "95581f20a76ec98d84f037aefdd78104"
server_key_version: "encryptionkey@0"
I20260808 02:08:10.832702 16029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:10.849323 16029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:10.852321 16029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:10.852530 16029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:10.854194 16029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:10.854283 16029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:10.854341 16029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:10.854372 16029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:08:10 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:10.866460 16029 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:39085
I20260808 02:08:10.866510 16220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:39085 every 8 connection(s)
I20260808 02:08:10.867581 16029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:10.869414 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:10.855386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.65:60463)
0808 02:08:10.855902 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:10.855945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:10.856795 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:10.857436 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:10.857448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:10.857986 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:10.858719 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:10.858744 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.859765 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.859769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:10.860945 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:10.860955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.861716 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.861721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:10.861958 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:10.862670 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:10.862695 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:10.864558 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:10.866633 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:10.866641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:10.866655 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:10.866911 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:10.867239 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:10.867242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:10.867243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:10.867371 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:10.867853 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:10.867859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:10.868166 (+ 307us) client_negotiation.cc:769] Sending connection context
0808 02:08:10.868369 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 02:08:10.868647 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":157,"threads_started":1}
I20260808 02:08:10.869781 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:10.854848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:60463 (local address 127.5.209.126:33859)
0808 02:08:10.854976 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:10.854987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:10.856049 (+ 1062us) server_negotiation.cc:407] Connection header received
0808 02:08:10.857004 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:10.857007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:10.857070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:10.857177 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:10.858902 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.859635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.861072 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.861507 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.861628 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:10.864731 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:10.864735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:10.864737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:10.864763 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:10.866507 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.867021 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.867024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.867026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.867136 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.867472 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.867479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.867483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.867736 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:10.867838 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:10.869487 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0808 02:08:10.869627 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:08:10.870666 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 16029
I20260808 02:08:10.870805 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:10.871042 16221 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33859
I20260808 02:08:10.871167 5957 external_mini_cluster.cc:1428] Setting key 5579c22e321e15412306b5bd948583fc
I20260808 02:08:10.871219 16221 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:10.871791 16221 heartbeater.cc:511] Master 127.5.209.126:33859 requested a full tablet report, sending...
I20260808 02:08:10.873872 15930 ts_manager.cc:194] Registered new tserver with Master: cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085)
I20260808 02:08:10.875197 15930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:60463
WARNING: no policy specified for oryx/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:08:10.884725 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:10.875815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:58641 (local address 127.5.209.126:33859)
0808 02:08:10.875920 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:10.875923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:10.875933 (+ 10us) server_negotiation.cc:407] Connection header received
0808 02:08:10.875962 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:10.875964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:10.875995 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:10.876096 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:10.877876 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.878742 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.879644 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.880084 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.880220 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:10.881382 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:10.881388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:10.881395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:10.881430 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:10.883088 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.883509 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.883515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.883516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.883632 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:10.883917 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:10.883920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:10.883922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:10.884113 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:10.884197 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:10.884432 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:08:10.884560 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:08:10.884817 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:10.873366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.65:58641)
0808 02:08:10.873706 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:10.873724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:10.873817 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:10.877452 (+ 3635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:10.877455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:10.877649 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:10.877749 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:10.877754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.878901 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.878904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:10.879506 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:10.879514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:10.880208 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:10.880210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:10.880281 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:10.880830 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:10.880842 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:10.881203 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:10.883212 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:10.883214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:10.883216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:10.883406 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:10.883735 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:10.883738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:10.883739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:10.883820 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:10.884200 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:10.884202 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:10.884284 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 02:08:10.884364 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:08:10.884477 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
I20260808 02:08:10.905002 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33859
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:11.026382 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:11.026631 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:11.026687 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:11.030659 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:11.030712 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:11.030750 16233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:08:11.030781 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:11.036039 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:11.037076 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:11.038168 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:11.048107 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:11.048135 16244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:11.048107 16240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:11.048388 16233 server_base.cc:1034] running on GCE node
I20260808 02:08:11.048820 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:11.049511 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:11.050673 16233 hybrid_clock.cc:648] HybridClock initialized: now 1786154891050649 us; error 36 us; skew 500 ppm
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:11.053543 16233 init.cc:377] Logged in from keytab as oryx/127.5.209.66@KRBTEST.COM (short username oryx)
I20260808 02:08:11.054351 16233 webserver.cc:460] Webserver started at http://127.5.209.66:37399/ using document root <none> and password file <none>
I20260808 02:08:11.054876 16233 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:11.054934 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:11.055097 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:11.057058 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "1a2fdc7e8c0f6096a352624bd7a69716"
server_key_iv: "e7c5b7bdd2cca20c5e72387121d924db"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.057646 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "1a2fdc7e8c0f6096a352624bd7a69716"
server_key_iv: "e7c5b7bdd2cca20c5e72387121d924db"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.061395 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:11.063743 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:11.064915 16233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:11.065011 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "1a2fdc7e8c0f6096a352624bd7a69716"
server_key_iv: "e7c5b7bdd2cca20c5e72387121d924db"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.065088 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:11.080899 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:11.084304 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:11.084460 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:11.086305 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:11.086371 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:11.086426 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:11.086457 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:11.099521 16233 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:46545
I20260808 02:08:11.099609 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:46545 every 8 connection(s)
I20260808 02:08:11.100804 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:11.103278 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.088906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:39145 (local address 127.5.209.126:33859)
0808 02:08:11.089072 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.089079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.089101 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:08:11.090786 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.090791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.090860 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.090953 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.092545 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.093835 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.095656 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.096079 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.096217 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.099396 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:11.099401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:11.099406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:11.099444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:11.101124 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.101818 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.101822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.101823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.101933 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.102273 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.102275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.102277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.102471 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:11.102555 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.103038 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.103151 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:08:11.103379 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 16233
I20260808 02:08:11.103497 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:11.103814 5957 external_mini_cluster.cc:1428] Setting key 3005f654a6254abc89784861fd8cbd3c
I20260808 02:08:11.104341 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.087686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.66:39145)
0808 02:08:11.088443 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.088544 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.090397 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.091358 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.091370 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.091800 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:11.092365 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.092389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.094006 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.094010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.095497 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.095514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.096267 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.096271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.096416 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.097060 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:11.097107 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:11.099209 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:11.101281 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.101289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.101306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.101681 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.102031 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.102033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.102035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.102176 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.102566 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:11.102571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:11.102808 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.102959 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.103315 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260808 02:08:11.105849 16413 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33859
I20260808 02:08:11.106025 16413 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:11.106645 16413 heartbeater.cc:511] Master 127.5.209.126:33859 requested a full tablet report, sending...
I20260808 02:08:11.107928 15931 ts_manager.cc:194] Registered new tserver with Master: 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545)
I20260808 02:08:11.108645 15931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:39145
I20260808 02:08:11.116158 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.108274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:48139 (local address 127.5.209.126:33859)
0808 02:08:11.108380 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.108384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.108793 (+ 409us) server_negotiation.cc:407] Connection header received
0808 02:08:11.109020 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.109025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.109078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.109129 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.109860 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.110381 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.111225 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.111450 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.111559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.112850 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:11.112853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:11.112855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:11.112876 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:11.114376 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.114875 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.114878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.114879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.114950 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.115257 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.115259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.115260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.115532 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:11.115615 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.115946 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.116037 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:08:11.116238 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.108209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.66:48139)
0808 02:08:11.108594 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.108706 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.108884 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.109298 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.109303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.109569 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:11.109737 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.109744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.110515 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.110518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.111102 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.111111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.111574 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.111578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.111674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.112199 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:11.112211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:11.112686 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:11.114495 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.114500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.114505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.114776 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.115048 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.115053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.115055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.115157 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.115635 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:11.115638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:11.115753 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.115882 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.116052 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
I20260808 02:08:11.137245 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33859
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:11.256407 16426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:11.256672 16426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:11.256757 16426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:11.261848 16426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:11.261905 16426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:11.261976 16426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:08:11.262039 16426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:11.267449 16426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.209.126:33859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:11.268510 16426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:11.269695 16426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:11.279994 16436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:11.280086 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:11.280138 16426 server_base.cc:1034] running on GCE node
W20260808 02:08:11.280020 16438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:11.280659 16426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:11.281277 16426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:11.282503 16426 hybrid_clock.cc:648] HybridClock initialized: now 1786154891282478 us; error 36 us; skew 500 ppm
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:11.285315 16426 init.cc:377] Logged in from keytab as oryx/127.5.209.67@KRBTEST.COM (short username oryx)
I20260808 02:08:11.286226 16426 webserver.cc:460] Webserver started at http://127.5.209.67:38811/ using document root <none> and password file <none>
I20260808 02:08:11.286810 16426 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:11.286903 16426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:11.287155 16426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:11.289276 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "0b59b628c469caaa695440042a97af50"
server_key_iv: "94d5cbef6fd6faa7521a3e5118c33089"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.289914 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "0b59b628c469caaa695440042a97af50"
server_key_iv: "94d5cbef6fd6faa7521a3e5118c33089"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.293975 16426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:11.296455 16447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:11.297792 16426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:11.297924 16426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
format_stamp: "Formatted at 2026-08-08 02:08:11 on dist-test-slave-bw88"
server_key: "0b59b628c469caaa695440042a97af50"
server_key_iv: "94d5cbef6fd6faa7521a3e5118c33089"
server_key_version: "encryptionkey@0"
I20260808 02:08:11.298045 16426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:11.323284 16426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:11.326750 16426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:11.326963 16426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:11.328469 16426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:11.328544 16426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:11.328620 16426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:11.328665 16426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:11.340940 16426 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:36957
I20260808 02:08:11.341012 16623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:36957 every 8 connection(s)
I20260808 02:08:11.341921 16426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:11.343766 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.329458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:49405 (local address 127.5.209.126:33859)
0808 02:08:11.329646 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.329652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.330510 (+ 858us) server_negotiation.cc:407] Connection header received
0808 02:08:11.331864 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.331870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.331944 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.332012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.333962 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.334727 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.336656 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.336953 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.337076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.339755 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:11.339760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:11.339763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:11.339795 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:11.341478 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.342016 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.342020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.342021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.342122 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.342560 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.342566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.342568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.342801 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:11.342909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.343474 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.343581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:08:11.344811 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.329816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.67:49405)
0808 02:08:11.330321 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.330381 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.331573 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.332490 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.332502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.332993 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:11.333769 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.333796 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.334869 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.334876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.336491 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.336505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.337151 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.337178 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.337436 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.338047 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:11.338064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:11.339564 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:11.341610 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.341617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.341631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.341894 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.342232 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.342235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.342237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.342419 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.342920 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:11.342925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:11.343239 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.343394 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.343695 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260808 02:08:11.344913 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 16426
I20260808 02:08:11.345059 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:11.345422 5957 external_mini_cluster.cc:1428] Setting key 21739c02ee43e080437e6a2e00bd857a
I20260808 02:08:11.346266 16624 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33859
I20260808 02:08:11.346405 16624 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:11.347000 16624 heartbeater.cc:511] Master 127.5.209.126:33859 requested a full tablet report, sending...
I20260808 02:08:11.348019 15931 ts_manager.cc:194] Registered new tserver with Master: c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:11.348789 15931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:49405
I20260808 02:08:11.355314 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.348597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:55891 (local address 127.5.209.126:33859)
0808 02:08:11.349119 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.349123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.349138 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:11.349173 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.349176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.349219 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.349278 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.349809 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.350522 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.351325 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.351526 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.351606 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.352686 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:11.352688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:11.352690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:11.352708 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:11.353982 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.354367 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.354369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.354371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.354440 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.354683 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.354685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.354686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.354846 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:11.354920 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.355128 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.355210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260808 02:08:11.355326 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.348523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.5.209.67:55891)
0808 02:08:11.348810 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.348822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.348930 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.349324 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.349327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.349562 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:11.349687 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.349693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.350657 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.350660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.351221 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.351228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.351640 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.351642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.351711 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.352251 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:11.352263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:11.352565 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:11.354081 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.354085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.354087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.354280 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.354519 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:11.354522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:11.354523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:11.354606 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:11.354927 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:11.354929 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:11.355006 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.355087 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.355196 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260808 02:08:11.359303 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:08:11.384732 16000 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:11.378821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.5.209.126:33859)
0808 02:08:11.378945 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.378949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.378961 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:11.379003 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.379005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.379044 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.379124 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.379812 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.380333 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.381143 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.381417 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.381495 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.384593 (+ 3098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 02:08:11 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:11.393920 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.386344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.5.209.126:33859)
0808 02:08:11.386460 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.386463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.386667 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:08:11.386743 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.386745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.386785 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.386851 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:11.387405 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.387886 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.388739 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.388938 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.389012 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.391207 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:11.391210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:11.391213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:11.391236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:11.392483 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.392856 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.392859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.392860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.392934 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:11.393169 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:11.393171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:11.393172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:11.393401 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:11.393509 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.393707 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.393802 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:08:11.396843 15931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:08:11.400012 15931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:11.415302 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.408803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:46545 (local address 127.0.0.1:36224)
0808 02:08:11.410238 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.410254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.410339 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.411140 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.411143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.411159 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:11.411237 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.411244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412777 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.412781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.413830 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.413841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.414743 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.414747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.414854 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.414893 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.415008 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.415138 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":791,"thread_start_us":124,"threads_started":1}
I20260808 02:08:11.415352 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.409010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:36957 (local address 127.0.0.1:43124)
0808 02:08:11.409900 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.409915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.410064 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.410872 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.410875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.410891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:11.410976 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.410985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412173 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.412180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.413799 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.413812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.415013 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.415017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.415099 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.415121 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.415182 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.415238 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":89,"spinlock_wait_cycles":14464,"thread_start_us":115,"threads_started":1}
I20260808 02:08:11.415585 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.409674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.5.209.67:36957)
0808 02:08:11.410394 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.410414 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.410440 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:08:11.410516 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.410521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.410720 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.411136 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:11.411279 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412041 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.413950 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.414822 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.415015 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.415193 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.415255 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.415398 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":140,"threads_started":1}
I20260808 02:08:11.415930 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.408961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.5.209.66:46545)
0808 02:08:11.410665 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.410681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.410705 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:08:11.410785 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.410789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.410963 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.411167 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:11.411580 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412621 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.413997 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.414608 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.414743 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.414926 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.415670 (+ 744us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.415777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":133,"threads_started":1}
I20260808 02:08:11.415958 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.408803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39085 (local address 127.0.0.1:43440)
0808 02:08:11.409835 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.409890 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.410064 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.410735 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.410741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.410763 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:11.410916 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.410923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412368 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.412371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.413799 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.413812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.415275 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.415281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.415378 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.415634 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.415718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.415807 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":104,"spinlock_wait_cycles":2816,"thread_start_us":77,"threads_started":1}
I20260808 02:08:11.416306 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.409113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.5.209.65:39085)
0808 02:08:11.409547 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.409555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.410237 (+ 682us) server_negotiation.cc:407] Connection header received
0808 02:08:11.410336 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.410342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.410570 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.410733 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:11.411053 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.412217 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.414059 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.414822 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.415019 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.415169 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.415803 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.416154 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":87,"threads_started":1}
I20260808 02:08:11.418641 16518 tablet_service.cc:1467] Processing CreateTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 (DEFAULT_TABLE table=test-table [id=4cc77d9815ab4725b727a6093ce9b18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.418744 16319 tablet_service.cc:1467] Processing CreateTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 (DEFAULT_TABLE table=test-table [id=4cc77d9815ab4725b727a6093ce9b18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.418740 16135 tablet_service.cc:1467] Processing CreateTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 (DEFAULT_TABLE table=test-table [id=4cc77d9815ab4725b727a6093ce9b18b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.419739 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d269c7b8a43f7b5bdd5a31854c910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.419786 16319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d269c7b8a43f7b5bdd5a31854c910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.419786 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00d269c7b8a43f7b5bdd5a31854c910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.427211 16661 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Bootstrap starting.
I20260808 02:08:11.428095 16663 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Bootstrap starting.
I20260808 02:08:11.428673 16664 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Bootstrap starting.
I20260808 02:08:11.429862 16661 tablet_bootstrap.cc:654] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.430475 16663 tablet_bootstrap.cc:654] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.430514 16664 tablet_bootstrap.cc:654] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.430892 16661 log.cc:824] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:11.431365 16663 log.cc:824] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:11.431380 16664 log.cc:824] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:11.433531 16663 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: No bootstrap required, opened a new log
I20260808 02:08:11.433555 16661 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: No bootstrap required, opened a new log
I20260808 02:08:11.433763 16663 ts_tablet_manager.cc:1397] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:08:11.433766 16661 ts_tablet_manager.cc:1397] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 02:08:11.433876 16664 tablet_bootstrap.cc:492] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: No bootstrap required, opened a new log
I20260808 02:08:11.434077 16664 ts_tablet_manager.cc:1397] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:08:11.437794 16663 raft_consensus.cc:348] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.438066 16663 raft_consensus.cc:374] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.438122 16663 raft_consensus.cc:729] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.438119 16664 raft_consensus.cc:348] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.438120 16661 raft_consensus.cc:348] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.438390 16661 raft_consensus.cc:374] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.438446 16661 raft_consensus.cc:729] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c1b4f115ee494e97fb1b856d43db6e, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.438390 16664 raft_consensus.cc:374] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.438560 16664 raft_consensus.cc:729] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.438622 16663 consensus_queue.cc:260] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.438892 16661 consensus_queue.cc:260] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.438985 16664 consensus_queue.cc:260] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.439697 16663 ts_tablet_manager.cc:1428] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:08:11.439994 16664 ts_tablet_manager.cc:1428] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:08:11.439994 16661 ts_tablet_manager.cc:1428] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:08:11.440109 16221 heartbeater.cc:503] Master 127.5.209.126:33859 was elected leader, sending a full tablet report...
I20260808 02:08:11.440399 16624 heartbeater.cc:503] Master 127.5.209.126:33859 was elected leader, sending a full tablet report...
I20260808 02:08:11.440691 16413 heartbeater.cc:503] Master 127.5.209.126:33859 was elected leader, sending a full tablet report...
W20260808 02:08:11.593051 16625 tablet.cc:2369] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:08:11.602226 16414 tablet.cc:2369] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:08:11.619160 16222 tablet.cc:2369] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:08:11.781574 16669 raft_consensus.cc:482] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:11.781814 16669 raft_consensus.cc:504] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.783008 16669 leader_election.cc:288] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:11.787299 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.783419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37969 (local address 127.5.209.66:46545)
0808 02:08:11.783582 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.783587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.783888 (+ 301us) server_negotiation.cc:407] Connection header received
0808 02:08:11.784056 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.784059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.784117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.784203 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:11.784536 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.785109 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786330 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.786722 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786864 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.786921 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.787110 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.787185 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:08:11.787328 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.783278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:46545 (local address 127.5.209.65:37969)
0808 02:08:11.783779 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.783792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.783944 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.784214 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.784217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.784260 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:11.784416 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.784422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.785229 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.785234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.786204 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.786214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.786863 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.786976 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.786995 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.787085 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.787154 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":241,"spinlock_wait_cycles":5504,"thread_start_us":111,"threads_started":1}
I20260808 02:08:11.787462 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.783437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:36745 (local address 127.5.209.67:36957)
0808 02:08:11.783579 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.783584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.783873 (+ 289us) server_negotiation.cc:407] Connection header received
0808 02:08:11.784052 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.784055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.784111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.784199 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:11.784559 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.785228 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786453 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.786860 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786992 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.787049 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.787254 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.787327 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:08:11.787503 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.783278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:36957 (local address 127.5.209.65:36745)
0808 02:08:11.783758 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:11.783771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:11.783920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:11.784212 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:11.784214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:11.784253 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:11.784434 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.784439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.785375 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.785378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.786319 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:11.786330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.786990 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.786994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:11.787110 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.787141 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:08:11.787232 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:08:11.787304 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":4,"spinlock_wait_cycles":15488,"thread_start_us":114,"threads_started":1}
I20260808 02:08:11.788184 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00d269c7b8a43f7b5bdd5a31854c910" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" is_pre_election: true
I20260808 02:08:11.788190 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00d269c7b8a43f7b5bdd5a31854c910" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" is_pre_election: true
I20260808 02:08:11.788502 16348 raft_consensus.cc:2454] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:11.788517 16545 raft_consensus.cc:2454] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:11.789031 16060 leader_election.cc:302] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5c1b4f115ee494e97fb1b856d43db6e, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:11.789376 16669 raft_consensus.cc:2781] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:11.789474 16669 raft_consensus.cc:482] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:11.789507 16669 raft_consensus.cc:3037] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:11.790633 16669 raft_consensus.cc:504] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.791005 16669 leader_election.cc:288] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Requested vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:11.791445 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00d269c7b8a43f7b5bdd5a31854c910" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
I20260808 02:08:11.791570 16348 raft_consensus.cc:3037] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:11.792618 16348 raft_consensus.cc:2454] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:11.792917 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00d269c7b8a43f7b5bdd5a31854c910" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
I20260808 02:08:11.792973 16061 leader_election.cc:302] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:11.793079 16545 raft_consensus.cc:3037] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:11.793190 16669 raft_consensus.cc:2781] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:11.793495 16669 raft_consensus.cc:686] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Becoming Leader. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Running, Role: LEADER
I20260808 02:08:11.793864 16669 consensus_queue.cc:237] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.794878 16545 raft_consensus.cc:2454] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:11.797678 15930 catalog_manager.cc:5495] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea9829756534da8a0a72f8528db57e9 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "cea9829756534da8a0a72f8528db57e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:11.868559 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:11.864704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.5.209.65:39085)
0808 02:08:11.864850 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:11.864854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:11.864868 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:08:11.864953 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:11.864955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:11.865013 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:11.865120 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:11.865398 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.865906 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.866843 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:11.867166 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:11.867275 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:11.867511 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:11.868068 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:11.868189 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:11.868332 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:08:11.868403 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:08:11.875378 16545 raft_consensus.cc:1261] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:11.875422 16348 raft_consensus.cc:1261] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:11.876117 16675 consensus_queue.cc:1035] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:11.876296 16669 consensus_queue.cc:1035] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:11.884079 16680 mvcc.cc:205] Tried to move back new op lower bound from 7316090437113618432 to 7316090436790583296. Current Snapshot: MvccSnapshot[applied={T|T < 7316090437113618432}]
I20260808 02:08:11.884840 16683 mvcc.cc:205] Tried to move back new op lower bound from 7316090437113618432 to 7316090436790583296. Current Snapshot: MvccSnapshot[applied={T|T < 7316090437113618432}]
I20260808 02:08:11.886540 16681 mvcc.cc:205] Tried to move back new op lower bound from 7316090437113618432 to 7316090436790583296. Current Snapshot: MvccSnapshot[applied={T|T < 7316090437113618432}]
I20260808 02:08:11.894856 15930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:11.895038 15930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:11.897785 15930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Sending DeleteTablet for 3 replicas of tablet d00d269c7b8a43f7b5bdd5a31854c910
I20260808 02:08:11.898623 16319 tablet_service.cc:1514] Processing DeleteTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:11 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260808 02:08:11.898617 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:11 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43440
I20260808 02:08:11.898815 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet d00d269c7b8a43f7b5bdd5a31854c910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:11 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43124
I20260808 02:08:11.899200 16694 tablet_replica.cc:330] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: stopping tablet replica
I20260808 02:08:11.899425 16693 tablet_replica.cc:330] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: stopping tablet replica
I20260808 02:08:11.899588 16694 raft_consensus.cc:2229] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:11.899883 16693 raft_consensus.cc:2229] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:11.900059 16694 raft_consensus.cc:2258] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:11.900014 15930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 02:08:11.900468 16693 raft_consensus.cc:2258] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 02:08:11.900486 15930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:11.900964 16695 tablet_replica.cc:330] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: stopping tablet replica
I20260808 02:08:11.901786 16694 ts_tablet_manager.cc:1905] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:11.902309 16695 raft_consensus.cc:2229] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:11.902622 16695 raft_consensus.cc:2258] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:11.904269 16695 ts_tablet_manager.cc:1905] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:11.906606 16135 tablet_service.cc:1467] Processing CreateTablet for tablet c6b7109deffd408187b179df6558723f (DEFAULT_TABLE table=test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.906919 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b7109deffd408187b179df6558723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.907261 16695 ts_tablet_manager.cc:1918] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:11.907361 16695 log.cc:1196] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/d00d269c7b8a43f7b5bdd5a31854c910
I20260808 02:08:11.907389 16518 tablet_service.cc:1467] Processing CreateTablet for tablet c6b7109deffd408187b179df6558723f (DEFAULT_TABLE table=test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.907691 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b7109deffd408187b179df6558723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.908002 16693 ts_tablet_manager.cc:1905] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:11.909502 16663 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Bootstrap starting.
I20260808 02:08:11.907680 16695 ts_tablet_manager.cc:1939] T d00d269c7b8a43f7b5bdd5a31854c910 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting consensus metadata
I20260808 02:08:11.909853 16661 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Bootstrap starting.
I20260808 02:08:11.910686 16663 tablet_bootstrap.cc:654] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.910955 16661 tablet_bootstrap.cc:654] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.912016 15909 catalog_manager.cc:4841] TS c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957): tablet d00d269c7b8a43f7b5bdd5a31854c910 (table test-table [id=4cc77d9815ab4725b727a6093ce9b18b]) successfully deleted
I20260808 02:08:11.912520 16661 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: No bootstrap required, opened a new log
I20260808 02:08:11.912621 16661 ts_tablet_manager.cc:1397] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:11.912632 16319 tablet_service.cc:1467] Processing CreateTablet for tablet c6b7109deffd408187b179df6558723f (DEFAULT_TABLE table=test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:11.912537 16693 ts_tablet_manager.cc:1918] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:11.913175 16661 raft_consensus.cc:348] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.913317 16661 raft_consensus.cc:374] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.913391 16661 raft_consensus.cc:729] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c1b4f115ee494e97fb1b856d43db6e, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.913460 16693 log.cc:1196] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/d00d269c7b8a43f7b5bdd5a31854c910
I20260808 02:08:11.912932 16319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b7109deffd408187b179df6558723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:11.913551 16661 consensus_queue.cc:260] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.913862 16661 ts_tablet_manager.cc:1428] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 02:08:11.913880 16693 ts_tablet_manager.cc:1939] T d00d269c7b8a43f7b5bdd5a31854c910 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting consensus metadata
I20260808 02:08:11.914404 16694 ts_tablet_manager.cc:1918] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:11.915609 16694 log.cc:1196] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/d00d269c7b8a43f7b5bdd5a31854c910
I20260808 02:08:11.915980 16694 ts_tablet_manager.cc:1939] T d00d269c7b8a43f7b5bdd5a31854c910 P cea9829756534da8a0a72f8528db57e9: Deleting consensus metadata
I20260808 02:08:11.916215 16664 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Bootstrap starting.
I20260808 02:08:11.916181 15910 catalog_manager.cc:4841] TS 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545): tablet d00d269c7b8a43f7b5bdd5a31854c910 (table test-table [id=4cc77d9815ab4725b727a6093ce9b18b]) successfully deleted
I20260808 02:08:11.917110 15914 catalog_manager.cc:4841] TS cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085): tablet d00d269c7b8a43f7b5bdd5a31854c910 (table test-table [id=4cc77d9815ab4725b727a6093ce9b18b]) successfully deleted
I20260808 02:08:11.917291 16664 tablet_bootstrap.cc:654] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:11.918273 16663 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: No bootstrap required, opened a new log
I20260808 02:08:11.918359 16663 ts_tablet_manager.cc:1397] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260808 02:08:11.918817 16663 raft_consensus.cc:348] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.918921 16663 raft_consensus.cc:374] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.918953 16663 raft_consensus.cc:729] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.919067 16663 consensus_queue.cc:260] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.919319 16663 ts_tablet_manager.cc:1428] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:08:11.919567 16664 tablet_bootstrap.cc:492] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: No bootstrap required, opened a new log
I20260808 02:08:11.919649 16664 ts_tablet_manager.cc:1397] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:11.920095 16664 raft_consensus.cc:348] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.920197 16664 raft_consensus.cc:374] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:11.920223 16664 raft_consensus.cc:729] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Initialized, Role: FOLLOWER
I20260808 02:08:11.920428 16664 consensus_queue.cc:260] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.920739 16664 ts_tablet_manager.cc:1428] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:08:11.994834 16691 raft_consensus.cc:482] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:11.995000 16691 raft_consensus.cc:504] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.995494 16691 leader_election.cc:288] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:11.995954 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b7109deffd408187b179df6558723f" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" is_pre_election: true
I20260808 02:08:11.995968 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b7109deffd408187b179df6558723f" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" is_pre_election: true
I20260808 02:08:11.996132 16348 raft_consensus.cc:2454] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:11.996133 16545 raft_consensus.cc:2454] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:11.996431 16060 leader_election.cc:302] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5c1b4f115ee494e97fb1b856d43db6e, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:11.996696 16691 raft_consensus.cc:2781] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:11.996766 16691 raft_consensus.cc:482] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:11.996825 16691 raft_consensus.cc:3037] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:11.998023 16691 raft_consensus.cc:504] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:11.998386 16691 leader_election.cc:288] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Requested vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:11.998816 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b7109deffd408187b179df6558723f" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
I20260808 02:08:11.998966 16348 raft_consensus.cc:3037] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.000260 16348 raft_consensus.cc:2454] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:12.000569 16061 leader_election.cc:302] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.000566 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b7109deffd408187b179df6558723f" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
I20260808 02:08:12.000718 16545 raft_consensus.cc:3037] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.000777 16691 raft_consensus.cc:2781] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:12.000874 16691 raft_consensus.cc:686] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Becoming Leader. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Running, Role: LEADER
I20260808 02:08:12.001082 16691 consensus_queue.cc:237] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.002007 16545 raft_consensus.cc:2454] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:12.003124 15930 catalog_manager.cc:5495] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea9829756534da8a0a72f8528db57e9 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "cea9829756534da8a0a72f8528db57e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154890, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.126@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:12.049068 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.039164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33859 (local address 127.0.0.1:32926)
0808 02:08:12.039481 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.039499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.039604 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.039910 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.039912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.040326 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:12.040433 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.040439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.041574 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.041577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.042144 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.042152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.042782 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.042785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.042848 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.043445 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:12.043457 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:12.045134 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:12.047384 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:12.047393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:12.047399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:12.047679 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:12.047997 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:12.047999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:12.048000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:12.048079 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:12.048596 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:12.048602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:12.048713 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.048805 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.048924 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
I20260808 02:08:12.049243 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.039242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.5.209.126:33859)
0808 02:08:12.039527 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.039536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.039556 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:08:12.039710 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.039713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.039768 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.039866 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:12.040570 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.041468 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.042273 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.042665 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.042790 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.045293 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:12.045299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:12.045303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:12.045360 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:12.047265 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:12.047791 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:12.047795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:12.047797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:12.047899 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:12.048180 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:12.048184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:12.048186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:12.048479 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:12.048618 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.048912 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.049049 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":118,"threads_started":1}
I20260808 02:08:12.053718 15930 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:32926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:08:12.054517 15930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:12.060164 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 03738097430c44f0b1e65fa674088971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf4a32c8c89641ce8e1469bf85f49d5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:08:12.060294 16319 tablet_service.cc:1467] Processing CreateTablet for tablet 03738097430c44f0b1e65fa674088971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf4a32c8c89641ce8e1469bf85f49d5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:08:12.060621 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03738097430c44f0b1e65fa674088971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.060622 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 03738097430c44f0b1e65fa674088971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf4a32c8c89641ce8e1469bf85f49d5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:08:12.060753 16319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03738097430c44f0b1e65fa674088971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.061012 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03738097430c44f0b1e65fa674088971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.063546 16664 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: Bootstrap starting.
I20260808 02:08:12.063656 16663 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: Bootstrap starting.
I20260808 02:08:12.064481 16663 tablet_bootstrap.cc:654] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.064512 16664 tablet_bootstrap.cc:654] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.065511 16661 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: Bootstrap starting.
I20260808 02:08:12.066138 16664 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: No bootstrap required, opened a new log
I20260808 02:08:12.066234 16664 ts_tablet_manager.cc:1397] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:12.066565 16661 tablet_bootstrap.cc:654] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.066789 16663 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: No bootstrap required, opened a new log
I20260808 02:08:12.066737 16664 raft_consensus.cc:348] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.066852 16663 ts_tablet_manager.cc:1397] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:12.066854 16664 raft_consensus.cc:374] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.066895 16664 raft_consensus.cc:729] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.067046 16664 consensus_queue.cc:260] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.067263 16663 raft_consensus.cc:348] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.067404 16663 raft_consensus.cc:374] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.067438 16663 raft_consensus.cc:729] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.067368 16671 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.067525 16671 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.067369 16664 ts_tablet_manager.cc:1428] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:12.067584 16663 consensus_queue.cc:260] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.067884 16663 ts_tablet_manager.cc:1428] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:12.067836 16691 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.067975 16691 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.068663 16661 tablet_bootstrap.cc:492] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: No bootstrap required, opened a new log
I20260808 02:08:12.068756 16661 ts_tablet_manager.cc:1397] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:08:12.069244 16661 raft_consensus.cc:348] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.069437 16661 raft_consensus.cc:374] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.069484 16661 raft_consensus.cc:729] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c1b4f115ee494e97fb1b856d43db6e, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.070397 16661 consensus_queue.cc:260] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.070710 16670 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.070849 16670 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.071131 16661 ts_tablet_manager.cc:1428] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:12.156239 16671 raft_consensus.cc:482] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:12.156440 16671 raft_consensus.cc:504] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.157857 16671 leader_election.cc:288] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.162099 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.158262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:40831 (local address 127.5.209.67:36957)
0808 02:08:12.158460 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.158466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.158583 (+ 117us) server_negotiation.cc:407] Connection header received
0808 02:08:12.158837 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.158839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.158885 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.158971 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:12.159220 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.159713 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.160847 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.161513 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.161641 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.161700 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.161881 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.161953 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 02:08:12.162110 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.158101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:36957 (local address 127.5.209.66:40831)
0808 02:08:12.158484 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.158501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.158665 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.158985 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.158988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.159006 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:12.159107 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.159112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.159847 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.159851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.160717 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.160727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.161637 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.161640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.161707 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.161759 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.161844 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.161914 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":7,"spinlock_wait_cycles":6528,"thread_start_us":135,"threads_started":1}
I20260808 02:08:12.162212 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.158335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:44841 (local address 127.5.209.65:39085)
0808 02:08:12.158505 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.158511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.158537 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:08:12.158730 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.158735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.158796 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.158894 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:12.159168 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.159687 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.161008 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.161673 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.161790 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.161863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.162018 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.162091 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:08:12.162171 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.158090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39085 (local address 127.5.209.66:44841)
0808 02:08:12.158458 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.158474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.158585 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.158909 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.158911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.158929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:12.159043 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.159050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.159804 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.159808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.160872 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.160883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.161788 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.161794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.161897 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.161917 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.161990 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.162040 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":46,"thread_start_us":147,"threads_started":1}
I20260808 02:08:12.162755 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03738097430c44f0b1e65fa674088971" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" is_pre_election: true
I20260808 02:08:12.162925 16545 raft_consensus.cc:2454] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 0.
I20260808 02:08:12.163008 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03738097430c44f0b1e65fa674088971" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9829756534da8a0a72f8528db57e9" is_pre_election: true
I20260808 02:08:12.163292 16164 raft_consensus.cc:2454] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 0.
I20260808 02:08:12.163312 16254 leader_election.cc:302] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, c5c1b4f115ee494e97fb1b856d43db6e; no voters:
I20260808 02:08:12.163618 16671 raft_consensus.cc:2781] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:12.163687 16671 raft_consensus.cc:482] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:12.163726 16671 raft_consensus.cc:3037] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.165103 16671 raft_consensus.cc:504] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.165486 16671 leader_election.cc:288] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 election: Requested vote from peers cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.165817 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03738097430c44f0b1e65fa674088971" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9829756534da8a0a72f8528db57e9"
I20260808 02:08:12.165880 16545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03738097430c44f0b1e65fa674088971" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
I20260808 02:08:12.165959 16164 raft_consensus.cc:3037] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.166019 16545 raft_consensus.cc:3037] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.166934 16545 raft_consensus.cc:2454] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 1.
I20260808 02:08:12.166934 16164 raft_consensus.cc:2454] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 1.
I20260808 02:08:12.167258 16257 leader_election.cc:302] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.167428 16671 raft_consensus.cc:2781] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:12.167627 16671 raft_consensus.cc:686] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 LEADER]: Becoming Leader. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Running, Role: LEADER
I20260808 02:08:12.167920 16671 consensus_queue.cc:237] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.169106 16718 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: TxnStatusTablet state changed. Reason: New leader 56831a5bc8024d20ba4a9cdb15f7ca15. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.169234 16718 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:08:12.169703 16721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:08:12.169893 16721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:08:12.170548 15929 catalog_manager.cc:5495] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:12.192034 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.188167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.5.209.66:46545)
0808 02:08:12.188321 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.188326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.188395 (+ 69us) server_negotiation.cc:407] Connection header received
0808 02:08:12.188627 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.188635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.188693 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.188797 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.189066 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.189643 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.190518 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.190737 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.190858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.191222 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.191627 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.191757 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.191835 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.191889 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:08:12.192035 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.188044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:46545 (local address 127.0.0.1:36230)
0808 02:08:12.188327 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.188343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.188459 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.188811 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.188814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.188822 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:08:12.188931 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.188937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.189755 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.189760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.190384 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.190392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.190886 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.190890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.191007 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.191051 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.191732 (+ 681us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.191750 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.191825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.191911 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260808 02:08:12.194979 16545 raft_consensus.cc:1261] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Refusing update from remote peer 56831a5bc8024d20ba4a9cdb15f7ca15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.195194 16164 raft_consensus.cc:1261] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Refusing update from remote peer 56831a5bc8024d20ba4a9cdb15f7ca15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.195498 16671 consensus_queue.cc:1035] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.195716 16718 consensus_queue.cc:1035] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.197548 16691 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: TxnStatusTablet state changed. Reason: New leader 56831a5bc8024d20ba4a9cdb15f7ca15. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.197679 16691 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.198159 16670 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: TxnStatusTablet state changed. Reason: New leader 56831a5bc8024d20ba4a9cdb15f7ca15. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.198271 16670 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.199193 16671 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.199304 16671 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:08:12.200570 16718 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.200667 16718 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P 56831a5bc8024d20ba4a9cdb15f7ca15: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:08:12.201980 16691 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.202096 16691 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P cea9829756534da8a0a72f8528db57e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.203598 16670 tablet_replica.cc:439] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } } }
I20260808 02:08:12.203712 16670 tablet_replica.cc:442] T 03738097430c44f0b1e65fa674088971 P c5c1b4f115ee494e97fb1b856d43db6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:08:12.216707 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.213017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:46545 (local address 127.5.209.65:46829)
0808 02:08:12.213445 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.213458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.213581 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.213837 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.213841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.213849 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:08:12.213963 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.213969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.214700 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.214706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.215420 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.215429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.215907 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.215910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.216005 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.216047 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.216379 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.216386 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.216473 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.216556 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260808 02:08:12.216693 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:46829 (local address 127.5.209.66:46545)
0808 02:08:12.213267 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.213273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.213531 (+ 258us) server_negotiation.cc:407] Connection header received
0808 02:08:12.213696 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.213699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.213742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.213833 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.214079 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.214580 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.215570 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.215785 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.215893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.216212 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.216281 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.216359 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.216487 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.216536 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:08:12.223476 16545 raft_consensus.cc:1261] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.223608 16348 raft_consensus.cc:1261] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.223853 16669 consensus_queue.cc:1035] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.224076 16691 consensus_queue.cc:1035] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.246009 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.242395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:34715 (local address 127.5.209.65:39085)
0808 02:08:12.242523 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.242529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.242740 (+ 211us) server_negotiation.cc:407] Connection header received
0808 02:08:12.242900 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.242903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.242950 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.243034 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.243315 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.243840 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.244823 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.245084 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.245178 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.245532 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.245623 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.245704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.245795 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.245848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:08:12.246094 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.242324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39085 (local address 127.5.209.66:34715)
0808 02:08:12.242656 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.242669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.242791 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.243056 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.243058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.243066 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:08:12.243204 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.243210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.243951 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.243955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.244691 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.244702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.245189 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.245192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.245296 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.245386 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.245712 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.245721 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.245786 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.245871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":130,"threads_started":1}
I20260808 02:08:12.269665 15930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.269819 15930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.271842 15930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Sending DeleteTablet for 3 replicas of tablet c6b7109deffd408187b179df6558723f
I20260808 02:08:12.272497 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b7109deffd408187b179df6558723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43440
I20260808 02:08:12.272548 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b7109deffd408187b179df6558723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43124
I20260808 02:08:12.272722 16694 tablet_replica.cc:330] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: stopping tablet replica
I20260808 02:08:12.272753 16319 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b7109deffd408187b179df6558723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260808 02:08:12.272867 16694 raft_consensus.cc:2229] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:12.272954 16693 tablet_replica.cc:330] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: stopping tablet replica
I20260808 02:08:12.273077 16693 raft_consensus.cc:2229] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.273101 16694 raft_consensus.cc:2258] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.273236 16693 raft_consensus.cc:2258] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.273284 16695 tablet_replica.cc:330] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: stopping tablet replica
I20260808 02:08:12.273455 16695 raft_consensus.cc:2229] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.273610 16695 raft_consensus.cc:2258] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.274152 16694 ts_tablet_manager.cc:1905] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.274358 16695 ts_tablet_manager.cc:1905] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.277617 16695 ts_tablet_manager.cc:1918] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:08:12.277623 16693 ts_tablet_manager.cc:1905] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.277769 16695 log.cc:1196] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/c6b7109deffd408187b179df6558723f
I20260808 02:08:12.278129 16695 ts_tablet_manager.cc:1939] T c6b7109deffd408187b179df6558723f P c5c1b4f115ee494e97fb1b856d43db6e: Deleting consensus metadata
I20260808 02:08:12.278805 16694 ts_tablet_manager.cc:1918] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:08:12.278882 16694 log.cc:1196] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/c6b7109deffd408187b179df6558723f
I20260808 02:08:12.278980 15909 catalog_manager.cc:4841] TS c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957): tablet c6b7109deffd408187b179df6558723f (table test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]) successfully deleted
I20260808 02:08:12.279142 16694 ts_tablet_manager.cc:1939] T c6b7109deffd408187b179df6558723f P cea9829756534da8a0a72f8528db57e9: Deleting consensus metadata
I20260808 02:08:12.279865 15914 catalog_manager.cc:4841] TS cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085): tablet c6b7109deffd408187b179df6558723f (table test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]) successfully deleted
I20260808 02:08:12.280754 16693 ts_tablet_manager.cc:1918] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:08:12.280848 16693 log.cc:1196] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/c6b7109deffd408187b179df6558723f
I20260808 02:08:12.281160 16693 ts_tablet_manager.cc:1939] T c6b7109deffd408187b179df6558723f P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting consensus metadata
I20260808 02:08:12.281980 15910 catalog_manager.cc:4841] TS 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545): tablet c6b7109deffd408187b179df6558723f (table test-table [id=0f2e6ff6ba2c4cc5a36caa8e3b76ee79]) successfully deleted
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[15868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:08:12.309435 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:12.301825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.5.209.126:33859)
0808 02:08:12.302000 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.302005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.302194 (+ 189us) server_negotiation.cc:407] Connection header received
0808 02:08:12.302363 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.302366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.302455 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.302540 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:12.303536 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.304070 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.305448 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.305727 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.305873 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.309265 (+ 3392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45592: BlockingRecv error: recv got EOF from 127.0.0.1:45592 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[15868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM
I20260808 02:08:12.320087 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.311164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.5.209.126:33859)
0808 02:08:12.311296 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.311302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.311508 (+ 206us) server_negotiation.cc:407] Connection header received
0808 02:08:12.311638 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.311641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.311684 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.311773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:12.312357 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.312850 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.313708 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.313929 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.314031 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.316786 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:12.316789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:12.316792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:12.316814 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:12.318342 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:12.318854 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:12.318857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:12.318858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:12.318931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:12.319251 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:12.319254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:12.319255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:12.319417 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:12.319515 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.319840 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.319935 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:08:12.322995 15930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:08:12.323539 15930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:12.328776 16135 tablet_service.cc:1467] Processing CreateTablet for tablet ffad29220fa54f9bad290c4ea462ffec (DEFAULT_TABLE table=test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.328940 16518 tablet_service.cc:1467] Processing CreateTablet for tablet ffad29220fa54f9bad290c4ea462ffec (DEFAULT_TABLE table=test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.329005 16319 tablet_service.cc:1467] Processing CreateTablet for tablet ffad29220fa54f9bad290c4ea462ffec (DEFAULT_TABLE table=test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.329115 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffad29220fa54f9bad290c4ea462ffec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.329259 16319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffad29220fa54f9bad290c4ea462ffec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.329259 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffad29220fa54f9bad290c4ea462ffec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.331710 16663 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Bootstrap starting.
I20260808 02:08:12.331905 16661 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Bootstrap starting.
I20260808 02:08:12.332562 16663 tablet_bootstrap.cc:654] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.332902 16661 tablet_bootstrap.cc:654] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.333640 16664 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Bootstrap starting.
I20260808 02:08:12.334161 16663 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: No bootstrap required, opened a new log
I20260808 02:08:12.334245 16663 ts_tablet_manager.cc:1397] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:08:12.334590 16664 tablet_bootstrap.cc:654] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.334765 16663 raft_consensus.cc:348] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.334900 16663 raft_consensus.cc:374] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.334931 16663 raft_consensus.cc:729] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.335052 16663 consensus_queue.cc:260] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.335402 16663 ts_tablet_manager.cc:1428] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 02:08:12.337632 16661 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: No bootstrap required, opened a new log
I20260808 02:08:12.337752 16661 ts_tablet_manager.cc:1397] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 02:08:12.338160 16661 raft_consensus.cc:348] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.338289 16661 raft_consensus.cc:374] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.338343 16661 raft_consensus.cc:729] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c1b4f115ee494e97fb1b856d43db6e, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.338510 16661 consensus_queue.cc:260] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.339141 16664 tablet_bootstrap.cc:492] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: No bootstrap required, opened a new log
I20260808 02:08:12.339251 16664 ts_tablet_manager.cc:1397] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 02:08:12.339701 16664 raft_consensus.cc:348] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.339800 16664 raft_consensus.cc:374] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.339823 16664 raft_consensus.cc:729] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.339975 16664 consensus_queue.cc:260] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.340303 16661 ts_tablet_manager.cc:1428] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:12.342978 16669 raft_consensus.cc:482] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:12.343075 16669 raft_consensus.cc:504] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.343477 16669 leader_election.cc:288] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.344043 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffad29220fa54f9bad290c4ea462ffec" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" is_pre_election: true
I20260808 02:08:12.344216 16348 raft_consensus.cc:2454] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:12.344457 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffad29220fa54f9bad290c4ea462ffec" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" is_pre_election: true
I20260808 02:08:12.344570 16544 raft_consensus.cc:2454] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 0.
I20260808 02:08:12.344558 16061 leader_election.cc:302] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.345219 16669 raft_consensus.cc:2781] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:12.345281 16669 raft_consensus.cc:482] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:12.345319 16669 raft_consensus.cc:3037] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.346773 16669 raft_consensus.cc:504] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.347141 16669 leader_election.cc:288] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Requested vote from peers 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.347626 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffad29220fa54f9bad290c4ea462ffec" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
I20260808 02:08:12.347723 16544 raft_consensus.cc:3037] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.348853 16544 raft_consensus.cc:2454] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:12.349080 16348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffad29220fa54f9bad290c4ea462ffec" candidate_uuid: "cea9829756534da8a0a72f8528db57e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15"
I20260808 02:08:12.349195 16348 raft_consensus.cc:3037] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.349195 16060 leader_election.cc:302] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5c1b4f115ee494e97fb1b856d43db6e, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.349385 16669 raft_consensus.cc:2781] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:12.349450 16669 raft_consensus.cc:686] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Becoming Leader. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Running, Role: LEADER
I20260808 02:08:12.349575 16669 consensus_queue.cc:237] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.350381 16348 raft_consensus.cc:2454] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea9829756534da8a0a72f8528db57e9 in term 1.
I20260808 02:08:12.350431 16664 ts_tablet_manager.cc:1428] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260808 02:08:12.352345 15931 catalog_manager.cc:5495] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea9829756534da8a0a72f8528db57e9 (127.5.209.65). New cstate: current_term: 1 leader_uuid: "cea9829756534da8a0a72f8528db57e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:12.366652 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.363215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.5.209.65:39085)
0808 02:08:12.363335 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.363339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.363380 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:08:12.363519 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.363522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.363577 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.363657 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.363884 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.364674 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.365570 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.365924 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.366029 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.366261 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.366350 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.366427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.366503 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.366551 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:08:12.368666 16544 raft_consensus.cc:1261] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.368804 16348 raft_consensus.cc:1261] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Refusing update from remote peer cea9829756534da8a0a72f8528db57e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.369122 16669 consensus_queue.cc:1035] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.369272 16675 consensus_queue.cc:1035] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.376701 15931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.376817 15931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.378919 15931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Sending DeleteTablet for 3 replicas of tablet ffad29220fa54f9bad290c4ea462ffec
I20260808 02:08:12.379413 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet ffad29220fa54f9bad290c4ea462ffec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43440
I20260808 02:08:12.379458 16319 tablet_service.cc:1514] Processing DeleteTablet for tablet ffad29220fa54f9bad290c4ea462ffec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260808 02:08:12.379488 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet ffad29220fa54f9bad290c4ea462ffec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43124
I20260808 02:08:12.379622 16694 tablet_replica.cc:330] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: stopping tablet replica
I20260808 02:08:12.379652 16693 tablet_replica.cc:330] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: stopping tablet replica
I20260808 02:08:12.379786 16694 raft_consensus.cc:2229] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:12.379787 16693 raft_consensus.cc:2229] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.379928 16693 raft_consensus.cc:2258] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.379990 16694 raft_consensus.cc:2258] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.380126 16695 tablet_replica.cc:330] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: stopping tablet replica
I20260808 02:08:12.380282 16695 raft_consensus.cc:2229] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.380430 16695 raft_consensus.cc:2258] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.380748 16694 ts_tablet_manager.cc:1905] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.381286 15931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:08:12.381749 15931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:12.382625 16693 ts_tablet_manager.cc:1905] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.383348 16694 ts_tablet_manager.cc:1918] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:12.383414 16694 log.cc:1196] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/ffad29220fa54f9bad290c4ea462ffec
I20260808 02:08:12.383621 16694 ts_tablet_manager.cc:1939] T ffad29220fa54f9bad290c4ea462ffec P cea9829756534da8a0a72f8528db57e9: Deleting consensus metadata
I20260808 02:08:12.384354 15914 catalog_manager.cc:4841] TS cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085): tablet ffad29220fa54f9bad290c4ea462ffec (table test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]) successfully deleted
I20260808 02:08:12.385141 16693 ts_tablet_manager.cc:1918] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:12.385232 16693 log.cc:1196] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/ffad29220fa54f9bad290c4ea462ffec
I20260808 02:08:12.385496 16695 ts_tablet_manager.cc:1905] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.385555 16693 ts_tablet_manager.cc:1939] T ffad29220fa54f9bad290c4ea462ffec P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting consensus metadata
I20260808 02:08:12.386267 15910 catalog_manager.cc:4841] TS 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545): tablet ffad29220fa54f9bad290c4ea462ffec (table test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]) successfully deleted
I20260808 02:08:12.386972 16319 tablet_service.cc:1467] Processing CreateTablet for tablet 469cbde141304a88bedc67d487431983 (DEFAULT_TABLE table=test-table [id=764bbf7f1f974588b6aecde5257b77c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.387147 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 469cbde141304a88bedc67d487431983 (DEFAULT_TABLE table=test-table [id=764bbf7f1f974588b6aecde5257b77c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.387228 16319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469cbde141304a88bedc67d487431983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.387377 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469cbde141304a88bedc67d487431983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.387965 16695 ts_tablet_manager.cc:1918] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:12.388031 16695 log.cc:1196] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/ffad29220fa54f9bad290c4ea462ffec
I20260808 02:08:12.388302 16695 ts_tablet_manager.cc:1939] T ffad29220fa54f9bad290c4ea462ffec P c5c1b4f115ee494e97fb1b856d43db6e: Deleting consensus metadata
I20260808 02:08:12.388936 16664 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Bootstrap starting.
I20260808 02:08:12.389549 15909 catalog_manager.cc:4841] TS c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957): tablet ffad29220fa54f9bad290c4ea462ffec (table test-table [id=b7c5358bc9f44e4e96d5d37488c029a8]) successfully deleted
I20260808 02:08:12.389685 16661 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Bootstrap starting.
I20260808 02:08:12.389816 16664 tablet_bootstrap.cc:654] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.390561 16661 tablet_bootstrap.cc:654] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.391247 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 469cbde141304a88bedc67d487431983 (DEFAULT_TABLE table=test-table [id=764bbf7f1f974588b6aecde5257b77c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:12.391284 16664 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: No bootstrap required, opened a new log
I20260808 02:08:12.391396 16664 ts_tablet_manager.cc:1397] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:08:12.391532 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469cbde141304a88bedc67d487431983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:12.391871 16664 raft_consensus.cc:348] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.391990 16664 raft_consensus.cc:374] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.392032 16664 raft_consensus.cc:729] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.392144 16661 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: No bootstrap required, opened a new log
I20260808 02:08:12.392230 16661 ts_tablet_manager.cc:1397] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:12.392184 16664 consensus_queue.cc:260] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.392436 16664 ts_tablet_manager.cc:1428] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:12.392659 16661 raft_consensus.cc:348] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.392787 16661 raft_consensus.cc:374] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.392833 16661 raft_consensus.cc:729] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c1b4f115ee494e97fb1b856d43db6e, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.392966 16661 consensus_queue.cc:260] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.393285 16661 ts_tablet_manager.cc:1428] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:08:12.394093 16663 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: Bootstrap starting.
I20260808 02:08:12.394917 16663 tablet_bootstrap.cc:654] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:12.396266 16663 tablet_bootstrap.cc:492] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: No bootstrap required, opened a new log
I20260808 02:08:12.396376 16663 ts_tablet_manager.cc:1397] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:12.396754 16663 raft_consensus.cc:348] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.396868 16663 raft_consensus.cc:374] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:12.396912 16663 raft_consensus.cc:729] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9829756534da8a0a72f8528db57e9, State: Initialized, Role: FOLLOWER
I20260808 02:08:12.397043 16663 consensus_queue.cc:260] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.397332 16663 ts_tablet_manager.cc:1428] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 02:08:12.515723 16718 raft_consensus.cc:482] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:12.515868 16718 raft_consensus.cc:504] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.516388 16718 leader_election.cc:288] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.516835 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469cbde141304a88bedc67d487431983" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9829756534da8a0a72f8528db57e9" is_pre_election: true
I20260808 02:08:12.516954 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469cbde141304a88bedc67d487431983" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" is_pre_election: true
I20260808 02:08:12.517009 16164 raft_consensus.cc:2454] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 0.
I20260808 02:08:12.517155 16544 raft_consensus.cc:2454] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 0.
I20260808 02:08:12.517405 16257 leader_election.cc:302] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.517630 16718 raft_consensus.cc:2781] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:12.517714 16718 raft_consensus.cc:482] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:12.517771 16718 raft_consensus.cc:3037] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.518872 16718 raft_consensus.cc:504] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.519217 16718 leader_election.cc:288] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 election: Requested vote from peers cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085), c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957)
I20260808 02:08:12.519608 16164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469cbde141304a88bedc67d487431983" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9829756534da8a0a72f8528db57e9"
I20260808 02:08:12.519596 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469cbde141304a88bedc67d487431983" candidate_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c1b4f115ee494e97fb1b856d43db6e"
I20260808 02:08:12.519742 16544 raft_consensus.cc:3037] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.519742 16164 raft_consensus.cc:3037] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:12.520910 16164 raft_consensus.cc:2454] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 1.
I20260808 02:08:12.520910 16544 raft_consensus.cc:2454] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56831a5bc8024d20ba4a9cdb15f7ca15 in term 1.
I20260808 02:08:12.521219 16257 leader_election.cc:302] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56831a5bc8024d20ba4a9cdb15f7ca15, cea9829756534da8a0a72f8528db57e9; no voters:
I20260808 02:08:12.521422 16718 raft_consensus.cc:2781] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:12.521533 16718 raft_consensus.cc:686] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 LEADER]: Becoming Leader. State: Replica: 56831a5bc8024d20ba4a9cdb15f7ca15, State: Running, Role: LEADER
I20260808 02:08:12.521653 16718 consensus_queue.cc:237] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } }
I20260808 02:08:12.523381 15929 catalog_manager.cc:5495] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56831a5bc8024d20ba4a9cdb15f7ca15" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 46545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:12.556546 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.552892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.5.209.66:46545)
0808 02:08:12.553003 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.553008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.553095 (+ 87us) server_negotiation.cc:407] Connection header received
0808 02:08:12.553218 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.553220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.553276 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.553384 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.553698 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.554491 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.555366 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.555719 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.555821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.556114 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.556200 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.556270 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.556355 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.556410 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:08:12.565217 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.562457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:46545 (local address 127.5.209.66:59817)
0808 02:08:12.562571 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.562586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:12.562659 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:12.562879 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:12.562881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:12.562887 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:08:12.562964 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.562968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.563598 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.563601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.564154 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:12.564162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.564546 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.564549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:12.564607 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.564631 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.564914 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.564922 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:08:12.565002 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:08:12.565095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":3}
I20260808 02:08:12.565223 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.562512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:59817 (local address 127.5.209.66:46545)
0808 02:08:12.562642 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:12.562645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:12.562658 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:12.562748 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:12.562751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:12.562792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:12.562857 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:12.563054 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.563505 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.564256 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:12.564452 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:12.564525 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:12.564733 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:12.564794 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:12.564850 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:12.565023 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:08:12.565088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":53}
I20260808 02:08:12.571349 16544 raft_consensus.cc:1261] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Refusing update from remote peer 56831a5bc8024d20ba4a9cdb15f7ca15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.571434 16164 raft_consensus.cc:1261] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Refusing update from remote peer 56831a5bc8024d20ba4a9cdb15f7ca15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:12.571780 16776 consensus_queue.cc:1035] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c1b4f115ee494e97fb1b856d43db6e" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.572000 16671 consensus_queue.cc:1035] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea9829756534da8a0a72f8528db57e9" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:12.609387 15929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.609573 15929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:12.611931 15929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce77e531976c4f4eb3c9f5167d20f91a: Sending DeleteTablet for 3 replicas of tablet 469cbde141304a88bedc67d487431983
I20260808 02:08:12.612525 16319 tablet_service.cc:1514] Processing DeleteTablet for tablet 469cbde141304a88bedc67d487431983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:36224
I20260808 02:08:12.612582 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 469cbde141304a88bedc67d487431983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43440
I20260808 02:08:12.612727 16693 tablet_replica.cc:330] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: stopping tablet replica
I20260808 02:08:12.612749 16694 tablet_replica.cc:330] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9: stopping tablet replica
I20260808 02:08:12.612715 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet 469cbde141304a88bedc67d487431983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:12 UTC) from {username='oryx', principal='oryx/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43124
I20260808 02:08:12.612850 16694 raft_consensus.cc:2229] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.612851 16693 raft_consensus.cc:2229] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:12.612901 16695 tablet_replica.cc:330] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: stopping tablet replica
I20260808 02:08:12.612969 16694 raft_consensus.cc:2258] T 469cbde141304a88bedc67d487431983 P cea9829756534da8a0a72f8528db57e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.613013 16695 raft_consensus.cc:2229] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:12.613150 16695 raft_consensus.cc:2258] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.613196 16693 raft_consensus.cc:2258] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:12.613999 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 16029
I20260808 02:08:12.614065 16693 ts_tablet_manager.cc:1905] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.614243 16695 ts_tablet_manager.cc:1905] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:12.616807 16695 ts_tablet_manager.cc:1918] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:08:12.616882 16695 log.cc:1196] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/469cbde141304a88bedc67d487431983
I20260808 02:08:12.617173 16695 ts_tablet_manager.cc:1939] T 469cbde141304a88bedc67d487431983 P c5c1b4f115ee494e97fb1b856d43db6e: Deleting consensus metadata
I20260808 02:08:12.619769 15909 catalog_manager.cc:4841] TS c5c1b4f115ee494e97fb1b856d43db6e (127.5.209.67:36957): tablet 469cbde141304a88bedc67d487431983 (table test-table [id=764bbf7f1f974588b6aecde5257b77c6]) successfully deleted
I20260808 02:08:12.617282 16693 ts_tablet_manager.cc:1918] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:08:12.621039 16693 log.cc:1196] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/469cbde141304a88bedc67d487431983
I20260808 02:08:12.621392 16693 ts_tablet_manager.cc:1939] T 469cbde141304a88bedc67d487431983 P 56831a5bc8024d20ba4a9cdb15f7ca15: Deleting consensus metadata
I20260808 02:08:12.622079 15910 catalog_manager.cc:4841] TS 56831a5bc8024d20ba4a9cdb15f7ca15 (127.5.209.66:46545): tablet 469cbde141304a88bedc67d487431983 (table test-table [id=764bbf7f1f974588b6aecde5257b77c6]) successfully deleted
W20260808 02:08:12.631814 15914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:39085 (error 108)
I20260808 02:08:12.632194 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 16233
I20260808 02:08:12.637878 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:12.636533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39085 (local address 127.0.0.1:49988)
0808 02:08:12.637704 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:12.637784 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":895,"threads_started":1}
W20260808 02:08:12.638168 15914 catalog_manager.cc:4584] TS cea9829756534da8a0a72f8528db57e9 (127.5.209.65:39085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 469cbde141304a88bedc67d487431983: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39085: connect: Connection refused (error 111)
I20260808 02:08:12.645972 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 16426
I20260808 02:08:12.659044 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 15887
2026-08-08T02:08:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3721 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[16786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[16786](info): set up 2 sockets
Aug 08 02:08:12 dist-test-slave-bw88 krb5kdc[16786](info): commencing operation
krb5kdc: starting...
W20260808 02:08:13.865653 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:13 dist-test-slave-bw88 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:08:13.996881 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:40363
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:40363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:14.113286 16801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:14.113641 16801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:14.113747 16801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:14.117734 16801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:14.117807 16801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:14.117843 16801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:14.117899 16801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:14.117936 16801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:14.123167 16801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:40363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:40363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:14.124176 16801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:14.125213 16801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:14.134125 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:14.134146 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:14.134196 16801 server_base.cc:1034] running on GCE node
W20260808 02:08:14.134161 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:14.134855 16801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:14.135987 16801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:14.137164 16801 hybrid_clock.cc:648] HybridClock initialized: now 1786154894137144 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T02:08:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1486 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:14 dist-test-slave-bw88 krb5kdc[16819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:14 dist-test-slave-bw88 krb5kdc[16819](info): set up 2 sockets
Aug 08 02:08:14 dist-test-slave-bw88 krb5kdc[16819](info): commencing operation
krb5kdc: starting...
W20260808 02:08:15.372398 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:15 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:08:15.500476 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:41461
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:15.618768 16835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:15.619020 16835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:15.619060 16835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:15.622972 16835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:15.623030 16835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:15.623050 16835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:15.623067 16835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:15.623082 16835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:15.628265 16835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:41461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:15.629282 16835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:15.630291 16835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:15.639183 16844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:15.639180 16848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:15.639178 16846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:15.639405 16835 server_base.cc:1034] running on GCE node
I20260808 02:08:15.639976 16835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:15.641137 16835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:15.642338 16835 hybrid_clock.cc:648] HybridClock initialized: now 1786154895642322 us; error 57 us; skew 500 ppm
Aug 08 02:08:15 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154895, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:15.645308 16835 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:08:15.646346 16835 webserver.cc:460] Webserver started at http://127.5.209.126:46019/ using document root <none> and password file <none>
I20260808 02:08:15.646941 16835 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:15.647040 16835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:15.647285 16835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:15.649889 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "275098371a9b46e98aff552004d2e961"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "4eaf4f79d86c467beb4c3169057b62c1"
server_key_iv: "7ee7a3ab21ba621ba7237157754ec8e4"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.650476 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "275098371a9b46e98aff552004d2e961"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "4eaf4f79d86c467beb4c3169057b62c1"
server_key_iv: "7ee7a3ab21ba621ba7237157754ec8e4"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.654310 16835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:15.657037 16858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:15.658396 16835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:08:15.658567 16835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "275098371a9b46e98aff552004d2e961"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "4eaf4f79d86c467beb4c3169057b62c1"
server_key_iv: "7ee7a3ab21ba621ba7237157754ec8e4"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.658686 16835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:15.670089 16835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:15.673440 16835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:15.673648 16835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:15.681027 16835 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:41461
I20260808 02:08:15.681063 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:41461 every 8 connection(s)
I20260808 02:08:15.682024 16835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:15.684865 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:15.688264 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 16835
I20260808 02:08:15.688378 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:15.688681 5957 external_mini_cluster.cc:1428] Setting key 64856553f2466c51c1661b432f5148eb
I20260808 02:08:15.691254 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Bootstrap starting.
I20260808 02:08:15.693589 16930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:15.694350 16930 log.cc:824] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:15.696915 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: No bootstrap required, opened a new log
Aug 08 02:08:15 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:15.701260 16930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:15.701608 16930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:15.701735 16930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275098371a9b46e98aff552004d2e961, State: Initialized, Role: FOLLOWER
I20260808 02:08:15.702298 16930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:15.702461 16930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:15.702581 16930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:15.702692 16930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:15.703630 16930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:15.704002 16930 leader_election.cc:302] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275098371a9b46e98aff552004d2e961; no voters:
I20260808 02:08:15.704368 16930 leader_election.cc:288] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:15.704442 16939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:15.704660 16939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 LEADER]: Becoming Leader. State: Replica: 275098371a9b46e98aff552004d2e961, State: Running, Role: LEADER
I20260808 02:08:15.704890 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:15.690180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.5.209.126:41461)
0808 02:08:15.690658 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:15.690674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:15.690711 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:08:15.691337 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:15.691353 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:15.691747 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:15.692083 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:15.692929 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:15.694081 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:15.695159 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:15.695803 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:15.695974 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:15.698774 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:15.698782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:15.698802 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:15.698834 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:15.701550 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:15.702067 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:15.702073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:15.702079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:15.702224 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:15.702535 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:15.702539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:15.702545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:15.703061 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:15.703287 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:15.703577 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 02:08:15.703915 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260808 02:08:15.705024 16939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:15.705750 16930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:15.707770 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "275098371a9b46e98aff552004d2e961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } } }
I20260808 02:08:15.707894 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:15.708025 16941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275098371a9b46e98aff552004d2e961. Latest consensus state: current_term: 1 leader_uuid: "275098371a9b46e98aff552004d2e961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } } }
I20260808 02:08:15.708156 16941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:15.708307 16956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:15.711172 16956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:15.717087 16956 catalog_manager.cc:1347] Generated new cluster ID: 4563e734ddea49e29133032675b56da4
I20260808 02:08:15.717151 16956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:15.735493 16956 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:15.736747 16956 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:15.763329 16956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Generated new TSK 0
I20260808 02:08:15.764302 16956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:08:15.804445 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:15.924424 16967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:15.924685 16967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:15.924727 16967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:15.928773 16967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:15.928830 16967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:15.928886 16967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:15.934321 16967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:15.935395 16967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:15.936448 16967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:15.946614 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:15.946669 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:15.946702 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:15.946802 16967 server_base.cc:1034] running on GCE node
I20260808 02:08:15.947235 16967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:15.947855 16967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:15.949031 16967 hybrid_clock.cc:648] HybridClock initialized: now 1786154895949009 us; error 31 us; skew 500 ppm
Aug 08 02:08:15 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154895, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:15.951903 16967 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:08:15.952797 16967 webserver.cc:460] Webserver started at http://127.5.209.65:35795/ using document root <none> and password file <none>
I20260808 02:08:15.953474 16967 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:15.953572 16967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:15.953832 16967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:15.955936 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "75e4ecc3c50e438391cd9d21d2b949ab"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "81f16d0e847407ed208aca12d4f0ac72"
server_key_iv: "3c753637ed6d080312d8d7ff95fda6d0"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.956530 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "75e4ecc3c50e438391cd9d21d2b949ab"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "81f16d0e847407ed208aca12d4f0ac72"
server_key_iv: "3c753637ed6d080312d8d7ff95fda6d0"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.960700 16967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:15.963459 16986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:15.964692 16967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:15.964836 16967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "75e4ecc3c50e438391cd9d21d2b949ab"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "81f16d0e847407ed208aca12d4f0ac72"
server_key_iv: "3c753637ed6d080312d8d7ff95fda6d0"
server_key_version: "encryptionkey@0"
I20260808 02:08:15.964954 16967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:15.986092 16967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:15.988991 16967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:15.989216 16967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:15.989904 16967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:15.990795 16967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:15.990885 16967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:15.990985 16967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:15.991039 16967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.000420 16967 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:33591
I20260808 02:08:16.000447 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:33591 every 8 connection(s)
I20260808 02:08:16.001402 16967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:16.001843 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 16967
I20260808 02:08:16.002023 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:16.002384 5957 external_mini_cluster.cc:1428] Setting key abdb4724ae5e2dc70aa0e038feda8658
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154895, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:08:16.017295 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.003487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:40521 (local address 127.5.209.126:41461)
0808 02:08:16.003636 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.003642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.004809 (+ 1167us) server_negotiation.cc:407] Connection header received
0808 02:08:16.006811 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.006815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.006873 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.006980 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.008218 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.008774 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.010117 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.010386 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.010509 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.013166 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:16.013170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:16.013174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:16.013206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:16.014800 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.015344 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.015349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.015351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.015440 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.015926 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.015930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.015932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.016139 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:16.016255 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:16.016948 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 02:08:16.017104 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:08:16.018182 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.004008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:40521)
0808 02:08:16.004620 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.004683 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.006576 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.007172 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.007179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.007562 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.008056 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.008098 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.008912 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.008916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.009979 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.009989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.010566 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.010570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.010755 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.011418 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:16.011436 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:16.012982 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:16.014928 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.014934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.014952 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.015227 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.015554 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.015557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.015559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.015819 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.016256 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:16.016265 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:16.016661 (+ 396us) client_negotiation.cc:769] Sending connection context
0808 02:08:16.016905 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 02:08:16.017257 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":130,"threads_started":1}
I20260808 02:08:16.019469 17151 heartbeater.cc:347] Connected to a master server at 127.5.209.126:41461
I20260808 02:08:16.019747 17151 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:16.020287 17151 heartbeater.cc:511] Master 127.5.209.126:41461 requested a full tablet report, sending...
I20260808 02:08:16.021986 16880 ts_manager.cc:194] Registered new tserver with Master: 75e4ecc3c50e438391cd9d21d2b949ab (127.5.209.65:33591)
I20260808 02:08:16.023146 16880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:40521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:08:16.034459 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:16.154814 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:16.155073 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:16.155117 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:16.159088 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:16.159143 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:16.159200 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:16.164639 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:16.165841 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:16.166844 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:16.176975 17168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.176954 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.176946 17166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:16.177357 17157 server_base.cc:1034] running on GCE node
I20260808 02:08:16.177834 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:16.178553 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:16.179754 17157 hybrid_clock.cc:648] HybridClock initialized: now 1786154896179737 us; error 33 us; skew 500 ppm
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:16.182628 17157 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:08:16.183449 17157 webserver.cc:460] Webserver started at http://127.5.209.66:37359/ using document root <none> and password file <none>
I20260808 02:08:16.183984 17157 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:16.184042 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:16.184214 17157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:16.186283 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "ec62b2d3cf704c89a06d2179daf3ec9a"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "c7b1a2e39842493611a31d1e7d99e65e"
server_key_iv: "e8ba78235874feb91202dd36ca3113d5"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.186810 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "ec62b2d3cf704c89a06d2179daf3ec9a"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "c7b1a2e39842493611a31d1e7d99e65e"
server_key_iv: "e8ba78235874feb91202dd36ca3113d5"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.190407 17157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260808 02:08:16.192759 17177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.194032 17157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:16.194157 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "ec62b2d3cf704c89a06d2179daf3ec9a"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "c7b1a2e39842493611a31d1e7d99e65e"
server_key_iv: "e8ba78235874feb91202dd36ca3113d5"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.194265 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:16.206202 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:16.209650 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:16.209820 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:16.210341 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:16.211138 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:16.211184 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.211225 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:16.211246 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.220891 17157 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:39907
I20260808 02:08:16.220969 17340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:39907 every 8 connection(s)
I20260808 02:08:16.221982 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:16.232131 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 17157
I20260808 02:08:16.232311 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:16.232784 5957 external_mini_cluster.cc:1428] Setting key ed9b88c9b268631c3b89373457b3cc74
I20260808 02:08:16.237324 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.223982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:59919 (local address 127.5.209.126:41461)
0808 02:08:16.224136 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.224140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.224952 (+ 812us) server_negotiation.cc:407] Connection header received
0808 02:08:16.226455 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.226458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.226508 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.226599 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.227868 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.228396 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.229768 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.229992 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.230081 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.232793 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:16.232797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:16.232799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:16.232825 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:16.234632 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.235160 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.235167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.235172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.235280 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.235632 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.235639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.235644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.235971 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:16.236085 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:16.236945 (+ 860us) server_negotiation.cc:299] Negotiation successful
0808 02:08:16.237099 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:08:16.238339 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.224306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:59919)
0808 02:08:16.224778 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.224826 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.226236 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.226757 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.226765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.227190 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.227728 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.227743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.228533 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.228536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.229646 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.229657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.230161 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.230165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.230318 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.230904 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:16.230921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:16.232605 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:16.234755 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.234760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.234773 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.235037 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.235386 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.235388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.235389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.235528 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.236157 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:16.236169 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:16.236608 (+ 439us) client_negotiation.cc:769] Sending connection context
0808 02:08:16.236855 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 02:08:16.237287 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260808 02:08:16.239645 17341 heartbeater.cc:347] Connected to a master server at 127.5.209.126:41461
I20260808 02:08:16.239943 17341 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:16.240674 17341 heartbeater.cc:511] Master 127.5.209.126:41461 requested a full tablet report, sending...
I20260808 02:08:16.242013 16880 ts_manager.cc:194] Registered new tserver with Master: ec62b2d3cf704c89a06d2179daf3ec9a (127.5.209.66:39907)
I20260808 02:08:16.242620 16880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:59919
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:08:16.265254 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:16.379992 17346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:16.380278 17346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:16.380391 17346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:16.384438 17346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:16.384526 17346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:16.384618 17346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:16.389932 17346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:16.390990 17346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:16.392048 17346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:16.402040 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.402064 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.402156 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:16.402480 17346 server_base.cc:1034] running on GCE node
I20260808 02:08:16.402922 17346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:16.403591 17346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:16.404791 17346 hybrid_clock.cc:648] HybridClock initialized: now 1786154896404773 us; error 37 us; skew 500 ppm
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:16.407745 17346 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:08:16.408689 17346 webserver.cc:460] Webserver started at http://127.5.209.67:33211/ using document root <none> and password file <none>
I20260808 02:08:16.409267 17346 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:16.409382 17346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:16.409642 17346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:16.411701 17346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "e7ea3cc469324bf9a850d975584f3587"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "014b1e48b5c4d621b9a2dc3abc77a7ba"
server_key_iv: "78033197b706646d8bfed00176f905ee"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.412279 17346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "e7ea3cc469324bf9a850d975584f3587"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "014b1e48b5c4d621b9a2dc3abc77a7ba"
server_key_iv: "78033197b706646d8bfed00176f905ee"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.415959 17346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 02:08:16.418358 17366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.419512 17346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:16.419656 17346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "e7ea3cc469324bf9a850d975584f3587"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "014b1e48b5c4d621b9a2dc3abc77a7ba"
server_key_iv: "78033197b706646d8bfed00176f905ee"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.419763 17346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:16.430614 17346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:16.433575 17346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:16.433779 17346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:16.434381 17346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:16.435191 17346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:16.435266 17346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.435333 17346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:16.435386 17346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.444775 17346 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:44611
I20260808 02:08:16.444808 17521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:44611 every 8 connection(s)
I20260808 02:08:16.445714 17346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:16.452595 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 17346
I20260808 02:08:16.452742 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:16.453063 5957 external_mini_cluster.cc:1428] Setting key 2b6134629feefc0b9388f610965d8d90
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:16.459642 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.447472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:55325 (local address 127.5.209.126:41461)
0808 02:08:16.447635 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.447639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.448440 (+ 801us) server_negotiation.cc:407] Connection header received
0808 02:08:16.449856 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.449859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.449922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.450030 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.451185 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.451721 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.453265 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.453670 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.453779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.456616 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:16.456619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:16.456621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:16.456646 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:16.457912 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.458366 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.458369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.458370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.458447 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:16.458746 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:16.458748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:16.458749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:16.458914 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:16.458989 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:16.459422 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 02:08:16.459519 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:08:16.460492 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.447832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:55325)
0808 02:08:16.448292 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.448333 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.449638 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.450212 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.450219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.450565 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.451032 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.451046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.451879 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.451887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.453137 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.453150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.453832 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.453839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.454049 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.454710 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:16.454731 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:16.456454 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:16.458016 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.458020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.458032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.458276 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.458528 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:16.458530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:16.458532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:16.458665 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:16.459001 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:16.459005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:16.459220 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 02:08:16.459370 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 02:08:16.459650 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":133,"threads_started":1}
I20260808 02:08:16.461637 17522 heartbeater.cc:347] Connected to a master server at 127.5.209.126:41461
I20260808 02:08:16.461880 17522 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:16.462384 17522 heartbeater.cc:511] Master 127.5.209.126:41461 requested a full tablet report, sending...
I20260808 02:08:16.463505 16880 ts_manager.cc:194] Registered new tserver with Master: e7ea3cc469324bf9a850d975584f3587 (127.5.209.67:44611)
I20260808 02:08:16.464023 16880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:55325
I20260808 02:08:16.467047 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
Entry for principal oryx/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab.
WARNING: no policy specified for oryx/127.5.209.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
Entry for principal oryx/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab.
WARNING: no policy specified for oryx/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
Entry for principal oryx/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab.
I20260808 02:08:16.547761 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 16967
I20260808 02:08:16.561766 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 17157
I20260808 02:08:16.571504 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 17346
I20260808 02:08:16.583961 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 16835
I20260808 02:08:16.595387 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:41461
--webserver_interface=127.5.209.126
--webserver_port=46019
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:16.709794 17532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:16.710059 17532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:16.710100 17532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:16.714107 17532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:16.714157 17532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:16.714176 17532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:16.714193 17532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:16.714208 17532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:16.719327 17532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:41461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=46019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:16.720347 17532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:16.721441 17532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:16.729966 17545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.730024 17541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.730032 17542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:16.730305 17532 server_base.cc:1034] running on GCE node
I20260808 02:08:16.730849 17532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:16.731851 17532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:16.733009 17532 hybrid_clock.cc:648] HybridClock initialized: now 1786154896732997 us; error 26 us; skew 500 ppm
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:16.735816 17532 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:08:16.736733 17532 webserver.cc:460] Webserver started at http://127.5.209.126:46019/ using document root <none> and password file <none>
I20260808 02:08:16.737299 17532 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:16.737428 17532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:16.741271 17532 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:08:16.743435 17553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.744621 17532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:16.744751 17532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "275098371a9b46e98aff552004d2e961"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "4eaf4f79d86c467beb4c3169057b62c1"
server_key_iv: "7ee7a3ab21ba621ba7237157754ec8e4"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.745312 17532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:16.761093 17532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:16.764043 17532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:16.764256 17532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:16.771718 17532 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:41461
I20260808 02:08:16.771731 17625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:41461 every 8 connection(s)
I20260808 02:08:16.772816 17532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:16.777324 17627 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:08:16.779042 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Bootstrap starting.
I20260808 02:08:16.782804 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 17532
I20260808 02:08:16.783345 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:33591
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:41461
--webserver_port=35795
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:08:16.798213 17627 log.cc:824] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:16.806030 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:08:16.806450 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Bootstrap complete.
I20260808 02:08:16.811066 17627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:16.811483 17627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275098371a9b46e98aff552004d2e961, State: Initialized, Role: FOLLOWER
I20260808 02:08:16.812005 17627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:16.812130 17627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:16.812220 17627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:16.812325 17627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:08:16.814424 17627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:16.814775 17627 leader_election.cc:302] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275098371a9b46e98aff552004d2e961; no voters:
I20260808 02:08:16.815136 17627 leader_election.cc:288] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:08:16.815228 17632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:08:16.815411 17632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [term 2 LEADER]: Becoming Leader. State: Replica: 275098371a9b46e98aff552004d2e961, State: Running, Role: LEADER
I20260808 02:08:16.815724 17632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } }
I20260808 02:08:16.816438 17627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:16.816699 17632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275098371a9b46e98aff552004d2e961. Latest consensus state: current_term: 2 leader_uuid: "275098371a9b46e98aff552004d2e961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } } }
I20260808 02:08:16.816823 17632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:16.817265 17634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "275098371a9b46e98aff552004d2e961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275098371a9b46e98aff552004d2e961" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 41461 } } }
I20260808 02:08:16.817380 17638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:16.817467 17634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:16.820871 17638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:16.821591 17638 catalog_manager.cc:1259] Loaded cluster ID: 4563e734ddea49e29133032675b56da4
I20260808 02:08:16.821657 17638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:16.824214 17638 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:16.825978 17638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 275098371a9b46e98aff552004d2e961: Loaded TSK: 0
I20260808 02:08:16.827747 17638 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:08:16.906387 17629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:16.906666 17629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:16.906741 17629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:16.910795 17629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:16.910867 17629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:16.910954 17629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:16.916132 17629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.65.keytab
--principal=oryx/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=35795
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:16.917241 17629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:16.918320 17629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:16.929226 17658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.929265 17661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:16.929257 17659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:16.929590 17629 server_base.cc:1034] running on GCE node
I20260808 02:08:16.930154 17629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:16.930872 17629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:16.932058 17629 hybrid_clock.cc:648] HybridClock initialized: now 1786154896932034 us; error 44 us; skew 500 ppm
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154896, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:16.935039 17629 init.cc:377] Logged in from keytab as oryx/127.5.209.65@KRBTEST.COM (short username oryx)
I20260808 02:08:16.936054 17629 webserver.cc:460] Webserver started at http://127.5.209.65:35795/ using document root <none> and password file <none>
I20260808 02:08:16.936651 17629 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:16.936728 17629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:16.940877 17629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:16.943169 17673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.944500 17629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:08:16.944645 17629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "75e4ecc3c50e438391cd9d21d2b949ab"
format_stamp: "Formatted at 2026-08-08 02:08:15 on dist-test-slave-bw88"
server_key: "81f16d0e847407ed208aca12d4f0ac72"
server_key_iv: "3c753637ed6d080312d8d7ff95fda6d0"
server_key_version: "encryptionkey@0"
I20260808 02:08:16.945200 17629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:16.960254 17629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:16.963338 17629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:16.963551 17629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:16.964185 17629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:16.965057 17629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:16.965107 17629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.965149 17629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:16.965164 17629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:16.975050 17629 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:33591
I20260808 02:08:16.975081 17836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:33591 every 8 connection(s)
I20260808 02:08:16.976099 17629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:16.981391 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 17629
I20260808 02:08:16.981945 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:39907
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:41461
--webserver_port=37359
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:16.990265 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.978311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:50877)
0808 02:08:16.978823 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.978877 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.980133 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.981268 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.981280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.981825 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.982514 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.982538 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.983890 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.983893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.984801 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.984812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.985704 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.985708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.985898 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.986588 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:16.986607 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:16.989210 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
W20260808 02:08:16.990255 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:16.978350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:50877 (local address 127.5.209.126:41461)
0808 02:08:16.978982 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.978997 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.979037 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:08:16.980496 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.980507 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.980805 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.981089 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.982727 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.983724 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.984952 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.985519 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.985724 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.989480 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:50877: BlockingRecv error: recv got EOF from 127.5.209.65:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":151,"threads_started":1}
W20260808 02:08:16.990824 17837 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:41461 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.209.126:41461: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:16.996573 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.991330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:37831)
0808 02:08:16.991459 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.991475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.991557 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.991822 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.991825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.991996 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.992102 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.992107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.992884 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.992889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.993506 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.993515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.993982 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.993986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.994078 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.994597 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:16.994609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:16.996438 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:08:16.996850 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:16.991436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37831 (local address 127.5.209.126:41461)
0808 02:08:16.991553 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.991557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.991571 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:08:16.991660 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.991663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.991716 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.991802 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.992220 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.992765 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.993639 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.993862 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.993977 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:16.996570 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:37831: BlockingRecv error: recv got EOF from 127.5.209.65:37831 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:16 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:17.002426 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:16.997171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:52589)
0808 02:08:16.997299 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:16.997315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:16.997434 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:16.997670 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:16.997673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:16.997835 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:16.997921 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.997927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.998594 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.998597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.999451 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:16.999460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.999869 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.999872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:16.999961 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.000399 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.000419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.002306 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:08:17.002573 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:16.997217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:52589 (local address 127.5.209.126:41461)
0808 02:08:16.997321 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:16.997324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:16.997386 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:08:16.997523 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:16.997526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:16.997564 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:16.997631 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:16.998025 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.998495 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.999570 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:16.999769 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:16.999839 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.002431 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:52589: BlockingRecv error: recv got EOF from 127.5.209.65:52589 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 02:08:17.002856 17837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:08:17.105623 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:17.105923 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:17.106000 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:17.110121 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:17.110213 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:17.110316 17841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:17.115965 17841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.66.keytab
--principal=oryx/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:39907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=37359
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:17.117170 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:17.118351 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:17.128377 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:17.128391 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:17.128687 17841 server_base.cc:1034] running on GCE node
W20260808 02:08:17.128399 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:17.129220 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:17.129935 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:17.131134 17841 hybrid_clock.cc:648] HybridClock initialized: now 1786154897131114 us; error 30 us; skew 500 ppm
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154897, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:17.134158 17841 init.cc:377] Logged in from keytab as oryx/127.5.209.66@KRBTEST.COM (short username oryx)
I20260808 02:08:17.135175 17841 webserver.cc:460] Webserver started at http://127.5.209.66:37359/ using document root <none> and password file <none>
I20260808 02:08:17.135746 17841 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:17.135820 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:17.139997 17841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:08:17.142690 17863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.144027 17841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:17.144163 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "ec62b2d3cf704c89a06d2179daf3ec9a"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "c7b1a2e39842493611a31d1e7d99e65e"
server_key_iv: "e8ba78235874feb91202dd36ca3113d5"
server_key_version: "encryptionkey@0"
I20260808 02:08:17.144997 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:17.165184 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:17.168391 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:17.168562 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:17.169140 17841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:17.170068 17841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:17.170120 17841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.170161 17841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:17.170182 17841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.180155 17841 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:39907
I20260808 02:08:17.180176 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:39907 every 8 connection(s)
I20260808 02:08:17.181305 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:17.183941 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 17841
I20260808 02:08:17.184478 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:44611
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:41461
--webserver_port=33211
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:08:17.193177 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.183259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:58227 (local address 127.5.209.126:41461)
0808 02:08:17.183403 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.183407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.184315 (+ 908us) server_negotiation.cc:407] Connection header received
0808 02:08:17.185827 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.185834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.185908 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.186029 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.187227 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.187766 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.188951 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.189186 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.189300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.193071 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:58227: BlockingRecv error: recv got EOF from 127.5.209.66:58227 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:08:17.194108 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.183647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:58227)
0808 02:08:17.184143 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.184196 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.185608 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.186192 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.186199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.186582 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.187068 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.187083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.187904 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.187908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.188811 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.188821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.189379 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.189385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.189567 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.190182 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.190199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.192951 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
W20260808 02:08:17.194636 18026 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:41461 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.209.126:41461: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:17.200100 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.195079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:42113)
0808 02:08:17.195195 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.195211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.195284 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.195530 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.195533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.195699 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.195829 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.195835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.196550 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.196555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.197123 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.197132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.197647 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.197650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.197747 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.198246 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.198258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.199981 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:08:17.200235 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.195200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:42113 (local address 127.5.209.126:41461)
0808 02:08:17.195318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.195324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.195338 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:08:17.195385 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.195387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.195422 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.195517 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.195948 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.196432 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.197249 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.197529 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.197642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.200093 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:42113: BlockingRecv error: recv got EOF from 127.5.209.66:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:17.206274 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.200745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:53585)
0808 02:08:17.200898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.200911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.200989 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.201306 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.201309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.201490 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.201580 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.201586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.202292 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.202295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.203184 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.203196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.203612 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.203617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.203704 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.204155 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.204170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.206156 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260808 02:08:17.206477 18026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:08:17.206704 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.200834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:53585 (local address 127.5.209.126:41461)
0808 02:08:17.200948 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.200951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.200963 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:17.201120 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.201123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.201184 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.201267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.201710 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.202187 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.203302 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.203500 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.203575 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.206343 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:53585: BlockingRecv error: recv got EOF from 127.5.209.66:53585 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 02:08:17.308815 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:17.309113 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:17.309190 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:17.313274 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:17.313380 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:17.313510 18032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:17.319026 18032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/krb5kdc/oryx_127.5.209.67.keytab
--principal=oryx/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:44611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=33211
--tserver_master_addrs=127.5.209.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:17.320062 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:17.321110 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:17.331135 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:17.331147 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:17.331153 18037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:17.331480 18032 server_base.cc:1034] running on GCE node
I20260808 02:08:17.331879 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:17.332515 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:17.333703 18032 hybrid_clock.cc:648] HybridClock initialized: now 1786154897333681 us; error 42 us; skew 500 ppm
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154897, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:17.336695 18032 init.cc:377] Logged in from keytab as oryx/127.5.209.67@KRBTEST.COM (short username oryx)
I20260808 02:08:17.337625 18032 webserver.cc:460] Webserver started at http://127.5.209.67:33211/ using document root <none> and password file <none>
I20260808 02:08:17.338208 18032 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:17.338325 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:17.342211 18032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:08:17.344318 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.345682 18032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:17.345820 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "e7ea3cc469324bf9a850d975584f3587"
format_stamp: "Formatted at 2026-08-08 02:08:16 on dist-test-slave-bw88"
server_key: "014b1e48b5c4d621b9a2dc3abc77a7ba"
server_key_iv: "78033197b706646d8bfed00176f905ee"
server_key_version: "encryptionkey@0"
I20260808 02:08:17.346375 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:17.391328 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:17.394963 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:17.395190 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:17.395852 18032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:17.396755 18032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:17.396831 18032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.396909 18032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:17.396971 18032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:17.406852 18032 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:44611
I20260808 02:08:17.406891 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:44611 every 8 connection(s)
I20260808 02:08:17.407886 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:17.412853 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18032
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:08:17.419852 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.409756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:34597 (local address 127.5.209.126:41461)
0808 02:08:17.409913 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.409918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.410856 (+ 938us) server_negotiation.cc:407] Connection header received
0808 02:08:17.412147 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.412150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.412198 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.412275 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.413757 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.414410 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.415614 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.415834 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.415949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.419732 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:34597: BlockingRecv error: recv got EOF from 127.5.209.67:34597 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:08:17.420465 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.410218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:34597)
0808 02:08:17.410700 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.410744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.411939 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.412454 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.412461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.412853 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.413586 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.413609 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.414552 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.414576 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.415476 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.415485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.415991 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.415995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.416216 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.416836 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.416852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.419635 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
W20260808 02:08:17.420975 18212 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:41461 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.209.126:41461: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
I20260808 02:08:17.423604 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.413909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.5.209.126:41461)
0808 02:08:17.414181 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.414195 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.414207 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:08:17.414250 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.414252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.414287 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.414387 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.415151 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.415682 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.416916 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.417237 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.417383 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.418991 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:17.418997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:17.419011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:17.419038 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:17.421295 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:17.421829 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:17.421836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:17.421842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:17.421998 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:17.422313 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:17.422315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:17.422316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:17.422783 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:17.422971 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:17.423214 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 02:08:17.423472 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:17.426848 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.421471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:40251)
0808 02:08:17.421611 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.421625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.421715 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.421996 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.421998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.422171 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.422292 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.422299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.423214 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.423218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.423891 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.423901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.424422 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.424425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.424491 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.424954 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.424966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.426714 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 02:08:17.426936 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.421503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:40251 (local address 127.5.209.126:41461)
0808 02:08:17.421626 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.421629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.421673 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:08:17.421834 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.421837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.421882 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.421972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.422408 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.423095 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.424005 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.424315 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.424422 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.426820 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:40251: BlockingRecv error: recv got EOF from 127.5.209.67:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:17 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:17.432132 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:17.427567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:33369)
0808 02:08:17.427658 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:17.427674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:17.427751 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:17.427971 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:17.427974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:17.428120 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:17.428186 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.428191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.428878 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.428882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.429445 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:17.429453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.429898 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.429901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:17.429980 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.430441 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:17.430453 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:17.432036 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 02:08:17.432371 18212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:08:17.432571 17840 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:17.427633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:33369 (local address 127.5.209.126:41461)
0808 02:08:17.427772 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:17.427779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:17.427794 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:17.427829 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:17.427831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:17.427864 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:17.427956 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:17.428300 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.428768 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.429567 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:17.429787 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:17.429894 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:17.432188 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:33369: BlockingRecv error: recv got EOF from 127.5.209.67:33369 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:18.009626 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:18.003567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:56471)
0808 02:08:18.003866 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:18.003884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:18.004004 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:18.004247 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:18.004250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:18.004437 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:18.004560 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.004565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.005433 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.005438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.006305 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.006318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.006925 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.006928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.007003 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.007568 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:18.007585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:18.009499 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
W20260808 02:08:18.009910 18218 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:18.003735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:56471 (local address 127.5.209.126:41461)
0808 02:08:18.003969 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:18.003973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:18.003988 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:18.004079 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:18.004081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:18.004126 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:18.004205 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:18.004671 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.005227 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.006453 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.006809 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.006899 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.009768 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:56471: BlockingRecv error: recv got EOF from 127.5.209.65:56471 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:18.212941 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:18.207168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:50345)
0808 02:08:18.207471 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:18.207487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:18.207624 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:18.207908 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:18.207911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:18.208142 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:18.208243 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.208250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.209084 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.209088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.209704 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.209713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.210202 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.210205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.210295 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.210863 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:18.210883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:18.212747 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
W20260808 02:08:18.212975 18218 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:18.207245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:50345 (local address 127.5.209.126:41461)
0808 02:08:18.207418 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:18.207423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:18.207573 (+ 150us) server_negotiation.cc:407] Connection header received
0808 02:08:18.207734 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:18.207737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:18.207793 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:18.207881 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:18.208371 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.208952 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.209834 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.210065 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.210186 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.212871 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:50345: BlockingRecv error: recv got EOF from 127.5.209.66:50345 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:18 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:18.439041 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:18.433078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:53649)
0808 02:08:18.433471 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:18.433495 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:18.433635 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:18.433920 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:18.433923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:18.434146 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:18.434244 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.434250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.435039 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.435045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.435630 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:18.435638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.436149 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.436152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:18.436243 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.436798 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:18.436813 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:18.438904 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
W20260808 02:08:18.439184 18218 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:18.433241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:53649 (local address 127.5.209.126:41461)
0808 02:08:18.433427 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:18.433432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:18.433577 (+ 145us) server_negotiation.cc:407] Connection header received
0808 02:08:18.433757 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:18.433760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:18.433812 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:18.433893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:18.434364 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.434916 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.435776 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:18.436032 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:18.436144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:18.439055 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:53649: BlockingRecv error: recv got EOF from 127.5.209.67:53649 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:19.016237 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:19.010504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:50503)
0808 02:08:19.010791 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:19.010804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:19.010898 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:19.011168 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:19.011171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:19.011355 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:19.011456 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.011462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.012248 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.012251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.012820 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.012828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.013259 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.013262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.013326 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.014027 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:19.014040 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:19.016050 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
W20260808 02:08:19.016304 18223 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:19.010572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:50503 (local address 127.5.209.126:41461)
0808 02:08:19.010871 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:19.010875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:19.010889 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:08:19.010996 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:19.010998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:19.011046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:19.011129 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:19.011562 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.012112 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.012931 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.013152 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.013230 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.016182 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:50503: BlockingRecv error: recv got EOF from 127.5.209.65:50503 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:19.225318 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:19.213794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:34009)
0808 02:08:19.214098 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:19.214124 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:19.214226 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:19.214462 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:19.214464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:19.214649 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:19.214750 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.214755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.215472 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.215474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.216017 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.216023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.216523 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.216526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.216593 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.222939 (+ 6346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:19.222952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:19.225174 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":122,"threads_started":1}
W20260808 02:08:19.225512 18223 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:19.213894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:34009 (local address 127.5.209.126:41461)
0808 02:08:19.214063 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:19.214070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:19.214198 (+ 128us) server_negotiation.cc:407] Connection header received
0808 02:08:19.214309 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:19.214311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:19.214358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:19.214425 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:19.214849 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.215379 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.216123 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.216389 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.216514 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.225318 (+ 8804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:34009: BlockingRecv error: recv got EOF from 127.5.209.66:34009 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:19 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:19.445693 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:19.439796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:41955)
0808 02:08:19.440112 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:19.440135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:19.440246 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:19.440487 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:19.440489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:19.440692 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:19.440795 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.440801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.441644 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.441648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.442365 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:19.442376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.442982 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.442985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:19.443048 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.443614 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:19.443626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:19.445543 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
W20260808 02:08:19.445818 18223 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:19.439975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:41955 (local address 127.5.209.126:41461)
0808 02:08:19.440139 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:19.440143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:19.440212 (+ 69us) server_negotiation.cc:407] Connection header received
0808 02:08:19.440327 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:19.440329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:19.440376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:19.440444 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:19.440895 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.441498 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.442513 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:19.442841 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:19.442963 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:19.445680 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:41955: BlockingRecv error: recv got EOF from 127.5.209.67:41955 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:20.023150 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:20.017213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:47513)
0808 02:08:20.017571 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:20.017589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:20.017788 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:20.018161 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:20.018166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:20.018385 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:20.018495 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.018502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.019294 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.019296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.019862 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.019869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.020354 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.020357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.020442 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.020977 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:20.020989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:20.022951 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":114,"threads_started":1}
W20260808 02:08:20.023195 18227 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:20.017282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:47513 (local address 127.5.209.126:41461)
0808 02:08:20.017637 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:20.017641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:20.017743 (+ 102us) server_negotiation.cc:407] Connection header received
0808 02:08:20.017900 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:20.017903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:20.017957 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:20.018070 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:20.018614 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.019197 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.019993 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.020231 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.020325 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.023071 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:47513: BlockingRecv error: recv got EOF from 127.5.209.65:47513 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":138,"threads_started":1}
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:20.231918 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:20.226171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:49983)
0808 02:08:20.226526 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:20.226539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:20.226676 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:20.226967 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:20.226970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:20.227197 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:20.227294 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.227300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.228086 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.228090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.228665 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.228673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.229294 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.229297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.229419 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.229937 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:20.229948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:20.231741 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":131,"threads_started":1}
W20260808 02:08:20.231954 18227 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:20.226251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:49983 (local address 127.5.209.126:41461)
0808 02:08:20.226416 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:20.226422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:20.226635 (+ 213us) server_negotiation.cc:407] Connection header received
0808 02:08:20.226790 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:20.226795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:20.226848 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:20.226939 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:20.227415 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.227949 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.228796 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.229078 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.229187 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.231858 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:49983: BlockingRecv error: recv got EOF from 127.5.209.66:49983 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:20 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:20.452181 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:20.446449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:36679)
0808 02:08:20.446739 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:20.446753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:20.446853 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:20.447079 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:20.447081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:20.447289 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:20.447397 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.447402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.448133 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.448135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.448814 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:20.448824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.449248 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.449250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:20.449311 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.450017 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:20.450038 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:20.452062 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
W20260808 02:08:20.452320 18227 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:20.446519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:36679 (local address 127.5.209.126:41461)
0808 02:08:20.446664 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:20.446668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:20.446819 (+ 151us) server_negotiation.cc:407] Connection header received
0808 02:08:20.446920 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:20.446922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:20.446971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:20.447039 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:20.447497 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.448028 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.448921 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:20.449145 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:20.449231 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:20.452201 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:36679: BlockingRecv error: recv got EOF from 127.5.209.67:36679 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:21.029321 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:21.023992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:37937)
0808 02:08:21.024284 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:21.024298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:21.024388 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:21.024628 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:21.024630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:21.024810 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:21.024909 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.024915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.025723 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.025726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.026284 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.026292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.026723 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.026725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.026784 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.027357 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:21.027369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:21.029203 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
W20260808 02:08:21.029563 18231 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:21.024058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37937 (local address 127.5.209.126:41461)
0808 02:08:21.024345 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:21.024348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:21.024362 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:08:21.024482 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:21.024485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:21.024531 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:21.024618 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:21.025005 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.025613 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.026388 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.026621 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.026703 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.029387 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:37937: BlockingRecv error: recv got EOF from 127.5.209.65:37937 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:21.238263 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:21.232772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:44993)
0808 02:08:21.233120 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:21.233134 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:21.233237 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:21.233503 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:21.233505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:21.233696 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:21.233833 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.233839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.234584 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.234587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.235168 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.235177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.235627 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.235631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.235707 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.236220 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:21.236232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:21.238144 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":144,"threads_started":1}
W20260808 02:08:21.238440 18231 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:21.232841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:44993 (local address 127.5.209.126:41461)
0808 02:08:21.232979 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:21.232983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:21.233201 (+ 218us) server_negotiation.cc:407] Connection header received
0808 02:08:21.233317 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:21.233319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:21.233390 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:21.233451 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:21.233938 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.234470 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.235277 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.235486 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.235562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.238280 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:44993: BlockingRecv error: recv got EOF from 127.5.209.66:44993 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:21 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:21.458595 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:21.452931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:56683)
0808 02:08:21.453230 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:21.453242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:21.453440 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:21.453736 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:21.453739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:21.453956 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:21.454054 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.454059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.454846 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.454851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.455419 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:21.455427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.455887 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.455894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:21.455981 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.456529 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:21.456543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:21.458482 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
W20260808 02:08:21.458897 18231 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:21.452995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:56683 (local address 127.5.209.126:41461)
0808 02:08:21.453160 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:21.453165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:21.453315 (+ 150us) server_negotiation.cc:407] Connection header received
0808 02:08:21.453548 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:21.453552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:21.453621 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:21.453710 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:21.454177 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.454728 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.455541 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:21.455772 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:21.455883 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:21.458628 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:56683: BlockingRecv error: recv got EOF from 127.5.209.67:56683 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:22.036036 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:22.030271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:54119)
0808 02:08:22.030630 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:22.030644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:22.030740 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:22.030988 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:22.030990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:22.031184 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:22.031283 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.031288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.032055 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.032058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.032893 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.032927 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.033475 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.033478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.033548 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.034125 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:22.034137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:22.035919 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":173,"threads_started":1}
W20260808 02:08:22.036252 18238 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:22.030374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:54119 (local address 127.5.209.126:41461)
0808 02:08:22.030625 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:22.030629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:22.030715 (+ 86us) server_negotiation.cc:407] Connection header received
0808 02:08:22.030821 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:22.030823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:22.030866 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:22.030945 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:22.031383 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.031922 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.033045 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.033318 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.033466 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.036097 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:54119: BlockingRecv error: recv got EOF from 127.5.209.65:54119 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:22.245992 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:22.239293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:41263)
0808 02:08:22.239614 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:22.239633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:22.239746 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:22.240126 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:22.240130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:22.240479 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:22.240644 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.240656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.241745 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.241749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.242385 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.242395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.242880 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.242884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.242966 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.243591 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:22.243605 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:22.245814 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
W20260808 02:08:22.246035 18238 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:22.239466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:41263 (local address 127.5.209.126:41461)
0808 02:08:22.239714 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:22.239718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:22.239733 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:08:22.239920 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:22.239924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:22.239986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:22.240074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:22.240758 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.241599 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.242540 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.242765 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.242844 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.245935 (+ 3091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:41263: BlockingRecv error: recv got EOF from 127.5.209.66:41263 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:22 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:22.465222 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:22.459655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:45025)
0808 02:08:22.459944 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:22.459962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:22.460080 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:22.460370 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:22.460373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:22.460597 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:22.460714 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.460720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.461591 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.461597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.462181 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:22.462190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.462679 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.462683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:22.462776 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.463273 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:22.463286 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:22.465103 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
W20260808 02:08:22.465440 18238 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:22.459697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:45025 (local address 127.5.209.126:41461)
0808 02:08:22.459829 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:22.459834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:22.460052 (+ 218us) server_negotiation.cc:407] Connection header received
0808 02:08:22.460185 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:22.460188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:22.460242 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:22.460340 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:22.460847 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.461480 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.462314 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:22.462556 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:22.462675 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:22.465249 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:45025: BlockingRecv error: recv got EOF from 127.5.209.67:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:23.042747 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:23.036902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:57125)
0808 02:08:23.037201 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:23.037215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:23.037315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:23.037609 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:23.037612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:23.037811 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:23.037920 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.037925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.038739 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.038742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.039392 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.039401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.039978 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.039981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.040075 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.040684 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:23.040697 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:23.042626 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20260808 02:08:23.042943 18244 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:23.036982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:57125 (local address 127.5.209.126:41461)
0808 02:08:23.037265 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:23.037269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:23.037285 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:23.037427 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:23.037430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:23.037476 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:23.037565 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:23.038038 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.038579 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.039487 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.039849 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.039957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.042767 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:57125: BlockingRecv error: recv got EOF from 127.5.209.65:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:23.252728 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:23.246981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:52141)
0808 02:08:23.247299 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:23.247312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:23.247411 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:23.247649 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:23.247651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:23.247840 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:23.247938 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.247942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.248857 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.248859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.249550 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.249561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.249992 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.249995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.250088 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.250699 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:23.250711 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:23.252606 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
W20260808 02:08:23.252868 18244 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:23.246998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:52141 (local address 127.5.209.126:41461)
0808 02:08:23.247140 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:23.247145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:23.247383 (+ 238us) server_negotiation.cc:407] Connection header received
0808 02:08:23.247488 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:23.247491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:23.247537 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:23.247623 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:23.248050 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.248751 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.249684 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.249903 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.249984 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.252743 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:52141: BlockingRecv error: recv got EOF from 127.5.209.66:52141 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:23 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:23.471827 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:23.466076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:47087)
0808 02:08:23.466408 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:23.466421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:23.466566 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:23.466879 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:23.466883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:23.467122 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:23.467224 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.467230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.468022 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.468027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.468615 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:23.468624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.469119 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.469123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:23.469220 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.469772 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:23.469785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:23.471650 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
W20260808 02:08:23.471891 18244 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:23.466158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:47087 (local address 127.5.209.126:41461)
0808 02:08:23.466312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:23.466317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:23.466502 (+ 185us) server_negotiation.cc:407] Connection header received
0808 02:08:23.466689 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:23.466693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:23.466752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:23.466859 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:23.467354 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.467902 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.468747 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:23.468982 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:23.469094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:23.471784 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:47087: BlockingRecv error: recv got EOF from 127.5.209.67:47087 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:24.049217 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:24.043557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:48059)
0808 02:08:24.043908 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:24.043922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:24.044028 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:24.044275 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:24.044277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:24.044466 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:24.044564 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.044570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.045326 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.045329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.045970 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.045979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.046406 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.046409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.046466 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.047007 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:24.047027 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:24.049094 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":97,"threads_started":1}
W20260808 02:08:24.049504 18252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:24.043650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:48059 (local address 127.5.209.126:41461)
0808 02:08:24.043912 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:24.043917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:24.044002 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:08:24.044112 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:24.044114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:24.044159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:24.044238 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:24.044669 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.045228 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.046080 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.046307 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.046389 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.049246 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:48059: BlockingRecv error: recv got EOF from 127.5.209.65:48059 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:24.264560 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:24.253520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:48061)
0808 02:08:24.253848 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:24.253861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:24.253962 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:24.254210 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:24.254212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:24.254396 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:24.254499 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.254504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.255323 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.255326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.255886 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.255894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.256373 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.256375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.256441 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.262529 (+ 6088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:24.262542 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:24.264440 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
W20260808 02:08:24.264796 18252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:24.253567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:48061 (local address 127.5.209.126:41461)
0808 02:08:24.253731 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:24.253735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:24.253947 (+ 212us) server_negotiation.cc:407] Connection header received
0808 02:08:24.254047 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:24.254049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:24.254097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:24.254178 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:24.254646 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.255210 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.256022 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.256265 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.256357 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.264681 (+ 8324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:48061: BlockingRecv error: recv got EOF from 127.5.209.66:48061 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:24 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:24.478546 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:24.472757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:55465)
0808 02:08:24.473081 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:24.473094 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:24.473201 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:24.473473 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:24.473476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:24.473661 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:24.473762 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.473768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.474493 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.474496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.475053 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:24.475061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.475468 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.475471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:24.475534 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.476168 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:24.476179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:24.478430 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
W20260808 02:08:24.478847 18252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:24.472824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:55465 (local address 127.5.209.126:41461)
0808 02:08:24.472971 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:24.472974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:24.473171 (+ 197us) server_negotiation.cc:407] Connection header received
0808 02:08:24.473283 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:24.473285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:24.473351 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:24.473424 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:24.473869 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.474391 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.475155 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:24.475368 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:24.475446 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:24.478582 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:55465: BlockingRecv error: recv got EOF from 127.5.209.67:55465 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:25.056049 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:25.050190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:35591)
0808 02:08:25.050467 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:25.050481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:25.050607 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:25.050943 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:25.050948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:25.051199 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:25.051307 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.051314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.052135 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.052138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.052686 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.052693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.053195 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.053199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.053290 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.053881 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:25.053893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:25.055919 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260808 02:08:25.056231 18259 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:25.050362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:35591 (local address 127.5.209.126:41461)
0808 02:08:25.050675 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:25.050679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:25.050692 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:25.050730 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:25.050732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:25.050776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:25.050866 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:25.051423 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.052027 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.052814 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.053065 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.053179 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.056055 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:35591: BlockingRecv error: recv got EOF from 127.5.209.65:35591 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:25.271486 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:25.265531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:45029)
0808 02:08:25.265834 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:25.265848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:25.265957 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:25.266211 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:25.266213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:25.266393 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:25.266498 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.266504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.267387 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.267389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.267970 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.267978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.268507 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.268511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.268604 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.269236 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:25.269251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:25.271306 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
W20260808 02:08:25.271528 18259 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:25.265592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:45029 (local address 127.5.209.126:41461)
0808 02:08:25.265750 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:25.265754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:25.265922 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:08:25.266041 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:25.266044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:25.266089 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:25.266170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:25.266607 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.267287 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.268091 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.268379 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.268499 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.271429 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:45029: BlockingRecv error: recv got EOF from 127.5.209.66:45029 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:25 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:25.484654 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:25.479319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:43607)
0808 02:08:25.479645 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:25.479659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:25.479766 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:25.479991 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:25.479993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:25.480183 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:25.480285 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.480290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.480991 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.480993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.481643 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:25.481652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.482054 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.482057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:25.482121 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.482686 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:25.482698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:25.484528 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":117,"threads_started":1}
W20260808 02:08:25.484786 18259 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:25.479364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:43607 (local address 127.5.209.126:41461)
0808 02:08:25.479511 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:25.479516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:25.479732 (+ 216us) server_negotiation.cc:407] Connection header received
0808 02:08:25.479845 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:25.479847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:25.479895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:25.479954 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:25.480381 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.480897 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.481745 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:25.481957 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:25.482035 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:25.484658 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:43607: BlockingRecv error: recv got EOF from 127.5.209.67:43607 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:26.062569 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:26.056946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:39435)
0808 02:08:26.057237 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:26.057251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:26.057388 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:26.057680 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:26.057682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:26.057873 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:26.057977 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.057983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.058729 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.058732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.059298 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.059306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.059730 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.059732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.059795 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.060292 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:26.060310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:26.062393 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20260808 02:08:26.062647 18264 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:26.056991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:39435 (local address 127.5.209.126:41461)
0808 02:08:26.057273 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:26.057276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:26.057294 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:08:26.057497 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:26.057500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:26.057547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:26.057643 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:26.058080 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.058625 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.059400 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.059631 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.059707 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.062517 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:39435: BlockingRecv error: recv got EOF from 127.5.209.65:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:26.278129 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:26.272261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:52505)
0808 02:08:26.272698 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:26.272711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:26.272855 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:26.273136 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:26.273139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:26.273378 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:26.273478 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.273484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.274301 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.274305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.274876 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.274884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.275363 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.275367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.275457 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.275972 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:26.275984 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:26.277918 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":113,"threads_started":1}
W20260808 02:08:26.278154 18264 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:26.272323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:52505 (local address 127.5.209.126:41461)
0808 02:08:26.272617 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:26.272622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:26.272798 (+ 176us) server_negotiation.cc:407] Connection header received
0808 02:08:26.272958 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:26.272962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:26.273015 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:26.273113 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:26.273606 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.274178 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.275000 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.275235 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.275346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.278059 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:52505: BlockingRecv error: recv got EOF from 127.5.209.66:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:26 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:26.491725 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:26.485657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:58955)
0808 02:08:26.486032 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:26.486052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:26.486235 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:26.486551 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:26.486555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:26.486827 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:26.486938 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.486945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.487730 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.487743 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.488317 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:26.488325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.488840 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.488843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:26.488933 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.489522 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:26.489535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:26.491584 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
W20260808 02:08:26.491865 18264 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:26.485786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:58955 (local address 127.5.209.126:41461)
0808 02:08:26.486003 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:26.486008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:26.486163 (+ 155us) server_negotiation.cc:407] Connection header received
0808 02:08:26.486357 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:26.486361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:26.486419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:26.486520 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:26.487065 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.487613 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.488472 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:26.488720 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:26.488833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:26.491720 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:58955: BlockingRecv error: recv got EOF from 127.5.209.67:58955 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:27.069593 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:27.063454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:56215)
0808 02:08:27.063748 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:27.063769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:27.063864 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:27.064131 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:27.064133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:27.064320 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:27.064424 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.064429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.065406 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.065410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.066286 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.066298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.066961 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.066964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.067055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.067542 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:27.067553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:27.069476 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
W20260808 02:08:27.069761 18268 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:27.063506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:56215 (local address 127.5.209.126:41461)
0808 02:08:27.063774 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:27.063777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:27.063822 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:08:27.063951 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:27.063953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:27.063996 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:27.064091 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:27.064528 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.065228 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.066406 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.066813 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.066927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.069622 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:56215: BlockingRecv error: recv got EOF from 127.5.209.65:56215 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:27.284608 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:27.278950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:32901)
0808 02:08:27.279262 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:27.279275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:27.279413 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:27.279704 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:27.279706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:27.279920 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:27.280018 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.280024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.280801 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.280806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.281418 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.281427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.281903 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.281906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.281993 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.282498 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:27.282509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:27.284415 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20260808 02:08:27.284759 18268 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:27.279015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:32901 (local address 127.5.209.126:41461)
0808 02:08:27.279183 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:27.279189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:27.279358 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:08:27.279522 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:27.279526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:27.279580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:27.279678 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:27.280146 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.280688 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.281543 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.281781 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.281896 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.284615 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:32901: BlockingRecv error: recv got EOF from 127.5.209.66:32901 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:27 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:27.498214 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:27.492645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:36555)
0808 02:08:27.492970 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:27.493012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:27.493167 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:27.493490 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:27.493492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:27.493687 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:27.493789 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.493794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.494576 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.494579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.495149 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:27.495157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.495569 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.495572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:27.495638 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.496129 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:27.496141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:27.498092 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260808 02:08:27.498365 18268 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:27.492793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:36555 (local address 127.5.209.126:41461)
0808 02:08:27.492967 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:27.492972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:27.493140 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:08:27.493256 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:27.493258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:27.493309 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:27.493464 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:27.493915 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.494471 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.495254 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:27.495468 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:27.495545 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:27.498241 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:36555: BlockingRecv error: recv got EOF from 127.5.209.67:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:28.075997 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:28.070535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:33367)
0808 02:08:28.070832 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:28.070846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:28.070950 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:28.071204 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:28.071207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:28.071396 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:28.071500 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.071506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.072242 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.072244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.072807 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.072815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.073227 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.073229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.073290 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.073911 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:28.073923 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:28.075800 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":103,"threads_started":1}
W20260808 02:08:28.076057 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:28.070589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:33367 (local address 127.5.209.126:41461)
0808 02:08:28.070840 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:28.070844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:28.070899 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:08:28.071036 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:28.071038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:28.071081 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:28.071160 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:28.071609 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.072143 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.072910 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.073127 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.073202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.075926 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:33367: BlockingRecv error: recv got EOF from 127.5.209.65:33367 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:28.290889 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:28.285645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:43253)
0808 02:08:28.285939 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:28.285952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:28.286058 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:28.286299 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:28.286301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:28.286488 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:28.286595 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.286600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.287308 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.287310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.287874 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.287882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.288312 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.288315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.288383 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.288873 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:28.288884 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:28.290772 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
W20260808 02:08:28.291023 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:28.285666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:43253 (local address 127.5.209.126:41461)
0808 02:08:28.285802 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:28.285806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:28.286021 (+ 215us) server_negotiation.cc:407] Connection header received
0808 02:08:28.286144 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:28.286147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:28.286191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:28.286259 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:28.286691 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.287212 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.287981 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.288205 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.288287 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.290905 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:43253: BlockingRecv error: recv got EOF from 127.5.209.66:43253 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:28 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:28.505147 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:28.499066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:52379)
0808 02:08:28.499374 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:28.499388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:28.499492 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:28.499801 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:28.499803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:28.499992 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:28.500106 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.500112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.500890 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.500893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.501503 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:28.501512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.501939 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.501941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:28.502002 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.502886 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:28.502902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:28.505031 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":125,"threads_started":1}
W20260808 02:08:28.505301 18273 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:28.499167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:52379 (local address 127.5.209.126:41461)
0808 02:08:28.499354 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:28.499359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:28.499467 (+ 108us) server_negotiation.cc:407] Connection header received
0808 02:08:28.499630 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:28.499633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:28.499687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:28.499767 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:28.500224 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.500792 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.501610 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:28.501832 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:28.501918 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:28.505179 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:52379: BlockingRecv error: recv got EOF from 127.5.209.67:52379 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:29.082691 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:29.076980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:38393)
0808 02:08:29.077271 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:29.077286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:29.077441 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:29.077718 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:29.077720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:29.077906 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:29.078015 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.078020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.078876 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.078888 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.079553 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.079563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.080017 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.080020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.080105 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.080611 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:29.080623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:29.082556 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
W20260808 02:08:29.082828 18279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:29.077066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:38393 (local address 127.5.209.126:41461)
0808 02:08:29.077358 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:29.077362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:29.077375 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:29.077539 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:29.077542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:29.077587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:29.077676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:29.078119 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.078731 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.079672 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.079891 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.079969 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.082698 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:38393: BlockingRecv error: recv got EOF from 127.5.209.65:38393 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:29.301478 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:29.291767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:43743)
0808 02:08:29.292052 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:29.292067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:29.292178 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:29.292445 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:29.292447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:29.292648 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:29.292754 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.292760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.293577 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.293579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.294138 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.294145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.294561 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.294564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.294645 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.299366 (+ 4721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:29.299379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:29.301300 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
W20260808 02:08:29.301755 18279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:29.291858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:43743 (local address 127.5.209.126:41461)
0808 02:08:29.292007 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:29.292011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:29.292147 (+ 136us) server_negotiation.cc:407] Connection header received
0808 02:08:29.292264 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:29.292266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:29.292310 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:29.292398 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:29.292858 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.293482 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.294241 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.294459 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.294537 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.301529 (+ 6992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:43743: BlockingRecv error: recv got EOF from 127.5.209.66:43743 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:29 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:29.512174 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:29.505948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:54857)
0808 02:08:29.506354 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:29.506372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:29.506506 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:29.506809 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:29.506812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:29.507028 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:29.507134 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.507140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.507920 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.507925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.508493 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:29.508502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.508990 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.508994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:29.509089 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.509778 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:29.509791 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:29.512047 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
W20260808 02:08:29.512347 18279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:29.506252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:54857 (local address 127.5.209.126:41461)
0808 02:08:29.506389 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:29.506397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:29.506433 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:08:29.506614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:29.506618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:29.506681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:29.506774 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:29.507259 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.507798 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.508628 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:29.508857 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:29.508969 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:29.512190 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:54857: BlockingRecv error: recv got EOF from 127.5.209.67:54857 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:30.089586 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:30.083597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:42143)
0808 02:08:30.083886 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:30.083900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:30.083992 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:30.084243 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:30.084246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:30.084435 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:30.084539 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.084544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.085550 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.085553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.086117 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.086125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.086583 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.086586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.086659 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.087167 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:30.087178 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:30.089465 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
W20260808 02:08:30.089815 18285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:30.083734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:42143 (local address 127.5.209.126:41461)
0808 02:08:30.084009 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:30.084013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:30.084026 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:30.084066 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:30.084069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:30.084114 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:30.084204 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:30.084667 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.085441 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.086222 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.086475 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.086568 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.089620 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:42143: BlockingRecv error: recv got EOF from 127.5.209.65:42143 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:30.308475 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:30.302303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:58745)
0808 02:08:30.302691 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:30.302710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:30.302878 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:30.303179 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:30.303182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:30.303409 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:30.303510 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.303516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.304318 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.304322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.304914 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.304923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.305440 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.305444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.305535 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.306131 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:30.306144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:30.308281 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
W20260808 02:08:30.308517 18285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:30.302505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:58745 (local address 127.5.209.126:41461)
0808 02:08:30.302693 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:30.302700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:30.302811 (+ 111us) server_negotiation.cc:407] Connection header received
0808 02:08:30.302987 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:30.302991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:30.303048 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:30.303150 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:30.303644 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.304197 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.305043 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.305284 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.305433 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.308410 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:58745: BlockingRecv error: recv got EOF from 127.5.209.66:58745 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:30 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:30.519204 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:30.513270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:59833)
0808 02:08:30.513608 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:30.513624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:30.513761 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:30.514078 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:30.514081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:30.514333 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:30.514480 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.514490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.515369 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.515372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.515954 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:30.515962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.516378 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.516381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:30.516445 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.516987 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:30.516999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:30.519060 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
W20260808 02:08:30.519351 18285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:30.513383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59833 (local address 127.5.209.126:41461)
0808 02:08:30.513515 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:30.513520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:30.513716 (+ 196us) server_negotiation.cc:407] Connection header received
0808 02:08:30.513869 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:30.513873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:30.513938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:30.514021 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:30.514599 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.515258 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.516054 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:30.516278 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:30.516354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:30.519201 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:59833: BlockingRecv error: recv got EOF from 127.5.209.67:59833 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:31.096350 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:31.090455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:60539)
0808 02:08:31.090743 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:31.090758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:31.090895 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:31.091222 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:31.091225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:31.091483 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:31.091646 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.091655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.092439 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.092441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.093106 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.093116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.093735 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.093739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.093829 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.094364 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:31.094376 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:31.096234 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260808 02:08:31.096550 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:31.090575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:60539 (local address 127.5.209.126:41461)
0808 02:08:31.090945 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:31.090949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:31.090962 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:08:31.091001 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:31.091003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:31.091067 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:31.091180 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:31.091781 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.092332 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.093228 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.093598 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.093716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.096376 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:60539: BlockingRecv error: recv got EOF from 127.5.209.65:60539 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":95,"threads_started":1}
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:31.315958 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:31.309592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:38881)
0808 02:08:31.309949 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:31.309963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:31.310067 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:31.310462 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:31.310465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:31.310673 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:31.310777 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.310782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.311614 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.311617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.312191 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.312199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.312721 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.312725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.312811 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.313521 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:31.313533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:31.315824 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260808 02:08:31.316174 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:31.309736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:38881 (local address 127.5.209.126:41461)
0808 02:08:31.309884 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:31.309888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:31.310034 (+ 146us) server_negotiation.cc:407] Connection header received
0808 02:08:31.310155 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:31.310157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:31.310328 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:31.310436 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:31.310914 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.311508 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.312301 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.312563 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.312668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.316032 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:38881: BlockingRecv error: recv got EOF from 127.5.209.66:38881 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:31 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.67@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:31.526428 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:31.520086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.67:34717)
0808 02:08:31.520403 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:31.520416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:31.520559 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:31.520861 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:31.520864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:31.521086 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:31.521208 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.521214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.522035 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.522039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.522647 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:31.522655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.523154 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.523157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:31.523298 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.523994 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:31.524006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:31.526239 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
W20260808 02:08:31.526468 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:31.520155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:34717 (local address 127.5.209.126:41461)
0808 02:08:31.520333 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:31.520338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:31.520495 (+ 157us) server_negotiation.cc:407] Connection header received
0808 02:08:31.520681 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:31.520684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:31.520742 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:31.520833 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:31.521364 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.521914 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.522773 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:31.523022 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:31.523134 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:31.526364 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.67:34717: BlockingRecv error: recv got EOF from 127.5.209.67:34717 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 02:08:32 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:32 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.65@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:32.103080 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:32.097390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.65:46855)
0808 02:08:32.097794 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:32.097808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:32.097907 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:32.098153 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:32.098156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:32.098351 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:32.098455 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.098460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.099197 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.099200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.099760 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.099768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.100272 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.100276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.100352 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.100932 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:32.100945 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:32.102901 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
W20260808 02:08:32.103178 18297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:32.097509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:46855 (local address 127.5.209.126:41461)
0808 02:08:32.097783 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:32.097787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:32.097864 (+ 77us) server_negotiation.cc:407] Connection header received
0808 02:08:32.097976 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:32.097978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:32.098028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:32.098128 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:32.098562 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.099089 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.099894 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.100143 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.100285 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.103031 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.65:46855: BlockingRecv error: recv got EOF from 127.5.209.65:46855 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
Aug 08 02:08:32 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:08:32 dist-test-slave-bw88 krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.209.66@KRBTEST.COM for oryx/127.5.209.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:08:32.323132 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:32.317002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:41461 (local address 127.5.209.66:57625)
0808 02:08:32.317306 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:32.317320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:32.317513 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:32.317859 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:32.317862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:32.318061 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:32.318164 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.318171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.318975 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.318978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.319674 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.319686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.320289 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.320294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.320427 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.321038 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:32.321051 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:32.323002 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.209.126:41461: Server oryx/127.5.209.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
W20260808 02:08:32.323323 18297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:32.317059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:57625 (local address 127.5.209.126:41461)
0808 02:08:32.317220 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:32.317224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:32.317427 (+ 203us) server_negotiation.cc:407] Connection header received
0808 02:08:32.317636 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:32.317639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:32.317690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:32.317762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:32.318277 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.318866 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.319805 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.320106 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.320243 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.323150 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.209.66:57625: BlockingRecv error: recv got EOF from 127.5.209.66:57625 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:08:32.423568 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 17629
I20260808 02:08:32.440529 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 17841
I20260808 02:08:32.453467 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18032
I20260808 02:08:32.464003 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 17532
2026-08-08T02:08:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18313 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T02:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:32Z Disabled control of system clock
I20260808 02:08:32.501237 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35381
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:08:32.615134 18302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:32.615427 18302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:32.615505 18302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:32.619520 18302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:32.619611 18302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:32.619647 18302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:32.619686 18302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:32.619719 18302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:32.624385 18302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:32.625567 18302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:32.626989 18302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:32.635937 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:32.636008 18302 server_base.cc:1034] running on GCE node
W20260808 02:08:32.635955 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:32.635950 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:32.636708 18302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:32.637907 18302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:32.639078 18302 hybrid_clock.cc:648] HybridClock initialized: now 1786154912639064 us; error 26 us; skew 500 ppm
I20260808 02:08:32.640959 18302 webserver.cc:460] Webserver started at http://127.5.209.126:43883/ using document root <none> and password file <none>
I20260808 02:08:32.641645 18302 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:32.641712 18302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:32.641958 18302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:32.644058 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "8149fdf8cd794fa691e074b0dd0d6ed4"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "ca212c7ee6118884cddc60352fe51d1c"
server_key_iv: "e889fff6ffe32f0ff79a347cdfb1f398"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.644670 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "8149fdf8cd794fa691e074b0dd0d6ed4"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "ca212c7ee6118884cddc60352fe51d1c"
server_key_iv: "e889fff6ffe32f0ff79a347cdfb1f398"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.648339 18302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:08:32.650841 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:32.652029 18302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:08:32.652172 18302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "8149fdf8cd794fa691e074b0dd0d6ed4"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "ca212c7ee6118884cddc60352fe51d1c"
server_key_iv: "e889fff6ffe32f0ff79a347cdfb1f398"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.652277 18302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:32.664705 18302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:32.665364 18302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:32.665658 18302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:32.672722 18302 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:35381
I20260808 02:08:32.672726 18389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:35381 every 8 connection(s)
I20260808 02:08:32.673719 18302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:32.676571 18390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:32.676946 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18302
I20260808 02:08:32.677100 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:32.677472 5957 external_mini_cluster.cc:1428] Setting key e00b0654cc3ba2aee7f64a1f05cf3736
I20260808 02:08:32.682986 18390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: Bootstrap starting.
I20260808 02:08:32.685680 18390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:32.686589 18390 log.cc:824] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:32.686887 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:32.678916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.5.209.126:35381)
0808 02:08:32.679544 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:32.679557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:32.679593 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:08:32.680246 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:32.680262 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:32.680767 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:32.681048 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:32.681460 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.682422 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.683667 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.684147 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.684271 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.685140 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:32.685145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:32.685158 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:32.685596 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:32.685651 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:32.685672 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:32.685777 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:32.685942 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:08:32.686154 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":135,"threads_started":1}
I20260808 02:08:32.688592 18390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: No bootstrap required, opened a new log
I20260808 02:08:32.692066 18390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } }
I20260808 02:08:32.692301 18390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:32.692385 18390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8149fdf8cd794fa691e074b0dd0d6ed4, State: Initialized, Role: FOLLOWER
I20260808 02:08:32.692817 18390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } }
I20260808 02:08:32.692970 18390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:32.693037 18390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:32.693174 18390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:32.694145 18390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } }
I20260808 02:08:32.694502 18390 leader_election.cc:302] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8149fdf8cd794fa691e074b0dd0d6ed4; no voters:
I20260808 02:08:32.694821 18390 leader_election.cc:288] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:32.694967 18395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:32.695228 18395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 8149fdf8cd794fa691e074b0dd0d6ed4, State: Running, Role: LEADER
I20260808 02:08:32.695616 18395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } }
I20260808 02:08:32.696141 18390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:32.697732 18396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } } }
I20260808 02:08:32.697873 18396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:32.698108 18397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8149fdf8cd794fa691e074b0dd0d6ed4. Latest consensus state: current_term: 1 leader_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149fdf8cd794fa691e074b0dd0d6ed4" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35381 } } }
I20260808 02:08:32.698225 18397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4 [sys.catalog]: This master's current role is: LEADER
W20260808 02:08:32.701138 18413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:08:32.701248 18413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:08:32.701330 18409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:32.702082 18409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:32.707569 18409 catalog_manager.cc:1347] Generated new cluster ID: a6c2c915a5c245a884836ce1c3c6b8fa
I20260808 02:08:32.707660 18409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:32.726670 18409 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:32.727895 18409 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:32.743445 18409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8149fdf8cd794fa691e074b0dd0d6ed4: Generated new TSK 0
I20260808 02:08:32.744305 18409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:08:32.748904 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--builtin_ntp_servers=127.5.209.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:08:32.866478 18419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:32.866791 18419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:32.866873 18419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:32.870932 18419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:32.871011 18419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:32.871110 18419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:32.875813 18419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:32.876951 18419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:32.878468 18419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:32.888447 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:32.888513 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:32.888744 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:32.889024 18419 server_base.cc:1034] running on GCE node
I20260808 02:08:32.889541 18419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:32.890223 18419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:32.891414 18419 hybrid_clock.cc:648] HybridClock initialized: now 1786154912891392 us; error 39 us; skew 500 ppm
I20260808 02:08:32.893329 18419 webserver.cc:460] Webserver started at http://127.5.209.65:43037/ using document root <none> and password file <none>
I20260808 02:08:32.893963 18419 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:32.894057 18419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:32.894280 18419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:32.896373 18419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "6b1f7b391f4643d185f3fe37ba211eb6"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "33993403e658d8e65eb68af2bb4d81ce"
server_key_iv: "545141307b3738d15d634efe3d67663f"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.897032 18419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1f7b391f4643d185f3fe37ba211eb6"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "33993403e658d8e65eb68af2bb4d81ce"
server_key_iv: "545141307b3738d15d634efe3d67663f"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.900842 18419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:08:32.903424 18439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:32.904628 18419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 02:08:32.904767 18419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "6b1f7b391f4643d185f3fe37ba211eb6"
format_stamp: "Formatted at 2026-08-08 02:08:32 on dist-test-slave-bw88"
server_key: "33993403e658d8e65eb68af2bb4d81ce"
server_key_iv: "545141307b3738d15d634efe3d67663f"
server_key_version: "encryptionkey@0"
I20260808 02:08:32.904879 18419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:32.916352 18419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:32.916997 18419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:32.917196 18419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:32.917830 18419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:32.918751 18419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:32.918823 18419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:32.918893 18419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:32.918936 18419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:32.927938 18419 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:42769
I20260808 02:08:32.927994 18592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:42769 every 8 connection(s)
I20260808 02:08:32.928825 18419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:32.936802 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18419
I20260808 02:08:32.936935 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:32.937315 5957 external_mini_cluster.cc:1428] Setting key 19b31e29cc72f2cc749ca0d89167abe4
I20260808 02:08:32.938536 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:32.930706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:51431 (local address 127.5.209.126:35381)
0808 02:08:32.930860 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:32.930866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:32.932015 (+ 1149us) server_negotiation.cc:407] Connection header received
0808 02:08:32.933439 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:32.933442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:32.933500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:32.933580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:32.934459 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.934983 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.936144 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.936525 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.936624 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.937992 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:32.937995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:32.937998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:32.938011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:32.938040 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:32.938048 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:32.938139 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:32.938315 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 02:08:32.938379 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:08:32.939445 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:32.931176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35381 (local address 127.5.209.65:51431)
0808 02:08:32.931858 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:32.931899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:32.933173 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:32.933760 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:32.933766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:32.933847 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:32.934310 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.934323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.935100 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.935102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.935978 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:32.935988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:32.936703 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:32.936707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:32.936931 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:32.937686 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:32.937688 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:32.937739 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:32.937824 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:32.938142 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:32.938155 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:32.938165 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:08:32.938301 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:08:32.938574 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
I20260808 02:08:32.939576 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--builtin_ntp_servers=127.5.209.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:08:32.940946 18593 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35381
I20260808 02:08:32.941237 18593 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:32.941924 18593 heartbeater.cc:511] Master 127.5.209.126:35381 requested a full tablet report, sending...
I20260808 02:08:32.943610 18342 ts_manager.cc:194] Registered new tserver with Master: 6b1f7b391f4643d185f3fe37ba211eb6 (127.5.209.65:42769)
I20260808 02:08:32.944800 18342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.65:51431
W20260808 02:08:33.058384 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:33.058663 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:33.058732 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:33.062781 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:33.062856 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:33.062968 18599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:33.068163 18599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:33.069257 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:33.070695 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:33.080691 18605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.080664 18611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.080687 18608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.080883 18599 server_base.cc:1034] running on GCE node
I20260808 02:08:33.081450 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:33.082141 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:33.083307 18599 hybrid_clock.cc:648] HybridClock initialized: now 1786154913083294 us; error 29 us; skew 500 ppm
I20260808 02:08:33.085537 18599 webserver.cc:460] Webserver started at http://127.5.209.66:39837/ using document root <none> and password file <none>
I20260808 02:08:33.086426 18599 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:33.086606 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:33.086910 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:33.089154 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "68d23e0f86004475824a4902bef1cc1a"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "58656fcd6c3444df9a5b2f544d709c19"
server_key_iv: "5a7b169ca48f1366a676caaaf7291fed"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.089931 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "68d23e0f86004475824a4902bef1cc1a"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "58656fcd6c3444df9a5b2f544d709c19"
server_key_iv: "5a7b169ca48f1366a676caaaf7291fed"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.095082 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:08:33.097954 18620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.099320 18599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:08:33.099459 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "68d23e0f86004475824a4902bef1cc1a"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "58656fcd6c3444df9a5b2f544d709c19"
server_key_iv: "5a7b169ca48f1366a676caaaf7291fed"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.099593 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:33.128253 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:33.129009 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:33.129223 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:33.129926 18599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:33.130914 18599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:33.131002 18599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.131107 18599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:33.131160 18599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.140013 18599 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:36463
I20260808 02:08:33.140055 18769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:36463 every 8 connection(s)
I20260808 02:08:33.140946 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:33.148458 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18599
I20260808 02:08:33.148552 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:33.148919 5957 external_mini_cluster.cc:1428] Setting key 724f45e7461e6ef5b071057e675ab633
I20260808 02:08:33.151314 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--builtin_ntp_servers=127.5.209.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:08:33.151705 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.142821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:52875 (local address 127.5.209.126:35381)
0808 02:08:33.142957 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:33.142961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:33.144056 (+ 1095us) server_negotiation.cc:407] Connection header received
0808 02:08:33.145543 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:33.145546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:33.145605 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:33.145696 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:33.146608 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.147149 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.148305 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.148571 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.148708 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:33.150086 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:33.150090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:33.150093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:33.150106 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:33.150128 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:33.150140 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:33.150229 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:33.150599 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 02:08:33.151591 (+ 992us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:08:33.151880 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.143213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35381 (local address 127.5.209.66:52875)
0808 02:08:33.143917 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:33.143956 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:33.145268 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:33.145882 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:33.145889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:33.145971 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:33.146458 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:33.146471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.147290 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.147294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:33.148176 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:33.148185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.148757 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.148762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:33.148979 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:33.149706 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:33.149708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:33.149762 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:33.149870 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:33.150401 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:33.150416 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:33.150426 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:08:33.150596 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 02:08:33.150876 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260808 02:08:33.153025 18770 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35381
I20260808 02:08:33.153282 18770 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:33.153939 18770 heartbeater.cc:511] Master 127.5.209.126:35381 requested a full tablet report, sending...
I20260808 02:08:33.155145 18342 ts_manager.cc:194] Registered new tserver with Master: 68d23e0f86004475824a4902bef1cc1a (127.5.209.66:36463)
I20260808 02:08:33.155979 18342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.66:52875
W20260808 02:08:33.270126 18774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:33.270375 18774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:33.270411 18774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:33.274308 18774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:33.274358 18774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:33.274406 18774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:33.278991 18774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:33.279933 18774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:33.281226 18774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:33.290940 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.291006 18774 server_base.cc:1034] running on GCE node
W20260808 02:08:33.290940 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.290961 18789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.291611 18774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:33.292245 18774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:33.293416 18774 hybrid_clock.cc:648] HybridClock initialized: now 1786154913293396 us; error 38 us; skew 500 ppm
I20260808 02:08:33.295176 18774 webserver.cc:460] Webserver started at http://127.5.209.67:39677/ using document root <none> and password file <none>
I20260808 02:08:33.295763 18774 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:33.295887 18774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:33.296098 18774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:33.298285 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "8fcfae02d8de4eeaad96aa67151e19e0"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "cbd3a954fef015be16fbeb3128bf25d0"
server_key_iv: "16f8957c74d33feb284284ab3b4906eb"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.298861 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "8fcfae02d8de4eeaad96aa67151e19e0"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "cbd3a954fef015be16fbeb3128bf25d0"
server_key_iv: "16f8957c74d33feb284284ab3b4906eb"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.302552 18774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:08:33.305254 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.306510 18774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:33.306658 18774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "8fcfae02d8de4eeaad96aa67151e19e0"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "cbd3a954fef015be16fbeb3128bf25d0"
server_key_iv: "16f8957c74d33feb284284ab3b4906eb"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.306777 18774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:33.322196 18774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:33.322975 18774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:33.323189 18774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:33.323809 18774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:33.324805 18774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:33.324882 18774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.324954 18774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:33.324998 18774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.334307 18774 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:37751
I20260808 02:08:33.334352 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:37751 every 8 connection(s)
I20260808 02:08:33.335248 18774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:33.339707 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18774
I20260808 02:08:33.339843 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:33.340200 5957 external_mini_cluster.cc:1428] Setting key e1f9837ed4da3f943cd1c11b02950ffa
I20260808 02:08:33.345088 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.337083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:44303 (local address 127.5.209.126:35381)
0808 02:08:33.337265 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:33.337270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:33.338386 (+ 1116us) server_negotiation.cc:407] Connection header received
0808 02:08:33.339961 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:33.339970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:33.340024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:33.340110 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:33.341304 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.341945 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.343202 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.343429 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.343543 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:33.344572 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:33.344576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:33.344579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:33.344592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:33.344620 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:33.344628 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:33.344709 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:33.344877 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:08:33.344938 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:08:33.345980 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.337525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:35381 (local address 127.5.209.67:44303)
0808 02:08:33.338227 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:33.338269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:33.339656 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:33.340329 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:33.340338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:33.340440 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:33.341134 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:33.341174 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.342078 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.342083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:33.343070 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:33.343080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:33.343593 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:33.343597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:33.343761 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:33.344318 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:33.344320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:33.344359 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:33.344421 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:33.344713 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:33.344725 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:33.344733 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:08:33.344874 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 02:08:33.345149 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260808 02:08:33.347028 18965 heartbeater.cc:347] Connected to a master server at 127.5.209.126:35381
I20260808 02:08:33.347246 18965 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:33.347700 18965 heartbeater.cc:511] Master 127.5.209.126:35381 requested a full tablet report, sending...
I20260808 02:08:33.348654 18342 ts_manager.cc:194] Registered new tserver with Master: 8fcfae02d8de4eeaad96aa67151e19e0 (127.5.209.67:37751)
I20260808 02:08:33.349171 18342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.67:44303
I20260808 02:08:33.354054 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:08:33.357318 18393 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:33.356267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.5.209.126:35381)
0808 02:08:33.356405 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:33.356409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:33.356603 (+ 194us) server_negotiation.cc:407] Connection header received
0808 02:08:33.356787 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:33.356790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:33.356840 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:33.356922 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:33.357206 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56806: BlockingRecv error: recv got EOF from 127.0.0.1:56806 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:08:33.357651 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18419
I20260808 02:08:33.376540 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18599
I20260808 02:08:33.393729 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18774
I20260808 02:08:33.406628 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18302
2026-08-08T02:08:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (941 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T02:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:33Z Disabled control of system clock
I20260808 02:08:33.442960 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34771
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:08:33.557267 18983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:33.557533 18983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:33.557600 18983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:33.561528 18983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:33.561576 18983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:33.561591 18983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:33.561640 18983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:33.561671 18983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:33.566289 18983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:33.567435 18983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:33.568893 18983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:33.577653 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.577708 18988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.577776 18983 server_base.cc:1034] running on GCE node
W20260808 02:08:33.577989 18994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.578542 18983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:33.579749 18983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:33.580933 18983 hybrid_clock.cc:648] HybridClock initialized: now 1786154913580914 us; error 33 us; skew 500 ppm
I20260808 02:08:33.582950 18983 webserver.cc:460] Webserver started at http://127.5.209.126:40529/ using document root <none> and password file <none>
I20260808 02:08:33.583570 18983 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:33.583668 18983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:33.583925 18983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:33.586086 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "85acbe5759284c7881b4c42155ba10f6"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "4ad985ada4f99491a546062e758a138c"
server_key_iv: "c9b3723c517c10c4cba8c2aeeaf2cba0"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.586670 18983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "85acbe5759284c7881b4c42155ba10f6"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "4ad985ada4f99491a546062e758a138c"
server_key_iv: "c9b3723c517c10c4cba8c2aeeaf2cba0"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.590492 18983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:08:33.592958 19003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.594257 18983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:08:33.594400 18983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "85acbe5759284c7881b4c42155ba10f6"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "4ad985ada4f99491a546062e758a138c"
server_key_iv: "c9b3723c517c10c4cba8c2aeeaf2cba0"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.594515 18983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:33.599179 18983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:33.599783 18983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:33.599972 18983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:33.607079 18983 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34771
I20260808 02:08:33.607102 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34771 every 8 connection(s)
I20260808 02:08:33.607975 18983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:33.608654 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 18983
I20260808 02:08:33.608786 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:33.609122 5957 external_mini_cluster.cc:1428] Setting key 60f3af878ed3bebb8f6c2c045fa039a6
I20260808 02:08:33.611655 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:33.615337 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.610635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.5.209.126:34771)
0808 02:08:33.611898 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:33.611915 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:33.611950 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:08:33.612600 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:33.612619 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:33.612953 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:33.613294 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:33.614106 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:33.614111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:33.614123 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:33.614133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:33.614197 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:33.614213 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:33.614364 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:33.614431 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 02:08:33.614666 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":725,"threads_started":1}
I20260808 02:08:33.617720 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6: Bootstrap starting.
I20260808 02:08:33.619871 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:33.620651 19072 log.cc:824] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:33.622736 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6: No bootstrap required, opened a new log
I20260808 02:08:33.626066 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } }
I20260808 02:08:33.626288 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:33.626322 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85acbe5759284c7881b4c42155ba10f6, State: Initialized, Role: FOLLOWER
I20260808 02:08:33.626722 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } }
I20260808 02:08:33.626880 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:33.626950 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:33.627144 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:33.628012 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } }
I20260808 02:08:33.628357 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85acbe5759284c7881b4c42155ba10f6; no voters:
I20260808 02:08:33.628669 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:33.628840 19078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:33.629092 19078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [term 1 LEADER]: Becoming Leader. State: Replica: 85acbe5759284c7881b4c42155ba10f6, State: Running, Role: LEADER
I20260808 02:08:33.629521 19078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } }
I20260808 02:08:33.629963 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:33.631310 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85acbe5759284c7881b4c42155ba10f6. Latest consensus state: current_term: 1 leader_uuid: "85acbe5759284c7881b4c42155ba10f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } } }
I20260808 02:08:33.631367 19080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85acbe5759284c7881b4c42155ba10f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85acbe5759284c7881b4c42155ba10f6" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34771 } } }
I20260808 02:08:33.631433 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:33.631470 19080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:33.631862 19089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:33.634761 19089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:33.640122 19089 catalog_manager.cc:1347] Generated new cluster ID: bcfab3157e1b41ae918f0223e3851318
I20260808 02:08:33.640209 19089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:33.656628 19089 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:33.657008 19089 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:33.661854 19089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85acbe5759284c7881b4c42155ba10f6: Generated new TSK 0
I20260808 02:08:33.662498 19089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:08:33.669510 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--builtin_ntp_servers=127.5.209.84:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:08:33.800107 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:33.800340 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:33.800403 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:33.804423 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:33.804472 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:33.804526 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:33.809118 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:33.810178 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:33.811563 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:33.821646 19115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.821671 19114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:33.821691 19117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:33.821831 19106 server_base.cc:1034] running on GCE node
I20260808 02:08:33.822422 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:33.823009 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:33.824172 19106 hybrid_clock.cc:648] HybridClock initialized: now 1786154913824153 us; error 29 us; skew 500 ppm
I20260808 02:08:33.825941 19106 webserver.cc:460] Webserver started at http://127.5.209.65:41923/ using document root <none> and password file <none>
I20260808 02:08:33.826455 19106 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:33.826517 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:33.826695 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:33.828619 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "16d9b96e5e054a699446c653014bdd2c"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "3c94db4c25c084e7fd53b06af8fb3724"
server_key_iv: "fec6a751914ce1332555b0d6e36ff9ad"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.829115 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "16d9b96e5e054a699446c653014bdd2c"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "3c94db4c25c084e7fd53b06af8fb3724"
server_key_iv: "fec6a751914ce1332555b0d6e36ff9ad"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.832947 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:33.835523 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.836714 19106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:33.836807 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "16d9b96e5e054a699446c653014bdd2c"
format_stamp: "Formatted at 2026-08-08 02:08:33 on dist-test-slave-bw88"
server_key: "3c94db4c25c084e7fd53b06af8fb3724"
server_key_iv: "fec6a751914ce1332555b0d6e36ff9ad"
server_key_version: "encryptionkey@0"
I20260808 02:08:33.836889 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:33.841270 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:33.841885 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:33.842020 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:33.842484 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:33.843308 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:33.843355 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.843391 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:33.843413 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:33.852350 19106 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:34815
I20260808 02:08:33.852371 19280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:34815 every 8 connection(s)
I20260808 02:08:33.853188 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:33.857718 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 19106
I20260808 02:08:33.857859 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:33.858170 5957 external_mini_cluster.cc:1428] Setting key 16bef1660feaaecdd7799a40d2d11d0e
I20260808 02:08:33.859843 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.855036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:40159 (local address 127.5.209.126:34771)
0808 02:08:33.855207 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:33.855212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:33.856347 (+ 1135us) server_negotiation.cc:407] Connection header received
0808 02:08:33.857720 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:33.857723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:33.857773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:33.857857 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:33.859343 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:33.859346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:33.859348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:33.859359 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:33.859380 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:33.859388 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:33.859485 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:33.859665 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 02:08:33.859712 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:08:33.860363 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--builtin_ntp_servers=127.5.209.84:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:08:33.860791 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:33.855466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34771 (local address 127.5.209.65:40159)
0808 02:08:33.856169 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:33.856215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:33.857487 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:33.858141 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:33.858148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:33.858253 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:33.859041 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:33.859043 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:33.859097 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:33.859183 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:33.859483 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:33.859495 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:33.859502 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:08:33.859704 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 02:08:33.859950 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":135,"threads_started":1}
I20260808 02:08:33.862109 19283 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34771
I20260808 02:08:33.862346 19283 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:33.862840 19283 heartbeater.cc:511] Master 127.5.209.126:34771 requested a full tablet report, sending...
I20260808 02:08:33.864462 19018 ts_manager.cc:194] Registered new tserver with Master: 16d9b96e5e054a699446c653014bdd2c (127.5.209.65:34815)
W20260808 02:08:33.978873 19288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:33.979120 19288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:33.979158 19288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:33.983753 19288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:33.983806 19288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:33.983896 19288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:33.988638 19288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:33.989655 19288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:33.991032 19288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:34.001070 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:34.001044 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:34.001029 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:34.001107 19288 server_base.cc:1034] running on GCE node
I20260808 02:08:34.001806 19288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:34.002455 19288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:34.003650 19288 hybrid_clock.cc:648] HybridClock initialized: now 1786154914003640 us; error 32 us; skew 500 ppm
I20260808 02:08:34.005479 19288 webserver.cc:460] Webserver started at http://127.5.209.66:34055/ using document root <none> and password file <none>
I20260808 02:08:34.006127 19288 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:34.006225 19288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:34.006465 19288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:34.008562 19288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "fec25a30b35f4f6fb4af9072a928adb0"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "ca2b3eb5eb15820568e34ef1cb77aaa5"
server_key_iv: "78955e5475e984b785d5c3d9c0f89d01"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.009192 19288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "fec25a30b35f4f6fb4af9072a928adb0"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "ca2b3eb5eb15820568e34ef1cb77aaa5"
server_key_iv: "78955e5475e984b785d5c3d9c0f89d01"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.012959 19288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:08:34.015708 19311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.016860 19288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:08:34.016996 19288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "fec25a30b35f4f6fb4af9072a928adb0"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "ca2b3eb5eb15820568e34ef1cb77aaa5"
server_key_iv: "78955e5475e984b785d5c3d9c0f89d01"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.017141 19288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:34.022557 19288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:34.023157 19288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:34.023343 19288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:34.023883 19288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:34.024804 19288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:34.024876 19288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.024947 19288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:34.024998 19288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.034267 19288 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:33485
I20260808 02:08:34.034345 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:33485 every 8 connection(s)
I20260808 02:08:34.035164 19288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:34.038534 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 19288
I20260808 02:08:34.038668 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:34.038964 5957 external_mini_cluster.cc:1428] Setting key e001149fc13fa82f42c964dbe15d808f
I20260808 02:08:34.041158 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--builtin_ntp_servers=127.5.209.84:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:08:34.041817 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:34.036975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:56529 (local address 127.5.209.126:34771)
0808 02:08:34.037161 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:34.037166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:34.038306 (+ 1140us) server_negotiation.cc:407] Connection header received
0808 02:08:34.039744 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:34.039747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:34.039806 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:34.039878 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:34.041189 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:34.041194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:34.041196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:34.041212 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:34.041242 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:34.041255 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:34.041392 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:34.041662 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:08:34.041708 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:08:34.042987 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:34.037457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34771 (local address 127.5.209.66:56529)
0808 02:08:34.038137 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:34.038176 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:34.039535 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:34.040077 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:34.040086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:34.040192 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:34.040894 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:34.040896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:34.040936 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:34.041005 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:34.041477 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:34.041492 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:34.041502 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:08:34.041687 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 02:08:34.041949 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
I20260808 02:08:34.044162 19479 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34771
I20260808 02:08:34.044421 19479 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:34.044855 19479 heartbeater.cc:511] Master 127.5.209.126:34771 requested a full tablet report, sending...
I20260808 02:08:34.045986 19018 ts_manager.cc:194] Registered new tserver with Master: fec25a30b35f4f6fb4af9072a928adb0 (127.5.209.66:33485)
W20260808 02:08:34.158684 19484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:34.158963 19484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:34.159067 19484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:34.163101 19484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:34.163178 19484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:34.163260 19484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:34.167995 19484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:34.169147 19484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:34.170622 19484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:34.180564 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:34.180548 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:34.180886 19484 server_base.cc:1034] running on GCE node
W20260808 02:08:34.180546 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:34.181435 19484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:34.182082 19484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:34.183251 19484 hybrid_clock.cc:648] HybridClock initialized: now 1786154914183218 us; error 49 us; skew 500 ppm
I20260808 02:08:34.185097 19484 webserver.cc:460] Webserver started at http://127.5.209.67:35123/ using document root <none> and password file <none>
I20260808 02:08:34.185711 19484 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:34.185775 19484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:34.185940 19484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:34.188052 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "5834f1d16c4e48e3a360c484514b085f"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "69ea6325ba6d064d8fb8c476fa24e639"
server_key_iv: "35af0a639fc767cfc230082effa224d9"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.188660 19484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "5834f1d16c4e48e3a360c484514b085f"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "69ea6325ba6d064d8fb8c476fa24e639"
server_key_iv: "35af0a639fc767cfc230082effa224d9"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.192307 19484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:08:34.194921 19504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.196071 19484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 02:08:34.196199 19484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "5834f1d16c4e48e3a360c484514b085f"
format_stamp: "Formatted at 2026-08-08 02:08:34 on dist-test-slave-bw88"
server_key: "69ea6325ba6d064d8fb8c476fa24e639"
server_key_iv: "35af0a639fc767cfc230082effa224d9"
server_key_version: "encryptionkey@0"
I20260808 02:08:34.196307 19484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:34.201278 19484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:34.201911 19484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:34.202095 19484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:34.202651 19484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:34.203570 19484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:34.203627 19484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.203696 19484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:34.203732 19484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:34.213004 19484 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:39647
I20260808 02:08:34.213044 19670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:39647 every 8 connection(s)
I20260808 02:08:34.213973 19484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:34.219123 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 19484
I20260808 02:08:34.219259 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:34.219605 5957 external_mini_cluster.cc:1428] Setting key 43c0490f90472c67a592ee5cd00ecc13
I20260808 02:08:34.220196 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:34.215786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:41047 (local address 127.5.209.126:34771)
0808 02:08:34.215947 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:34.215952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:34.217018 (+ 1066us) server_negotiation.cc:407] Connection header received
0808 02:08:34.218357 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:34.218360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:34.218400 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:34.218478 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:34.219679 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:34.219684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:34.219688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:08:34.219703 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:34.219728 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:08:34.219739 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:34.219834 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:34.220002 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:08:34.220046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:08:34.221370 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:34.216209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34771 (local address 127.5.209.67:41047)
0808 02:08:34.216869 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:34.216909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:34.218164 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:34.218645 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:34.218652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:34.218738 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:34.219392 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:08:34.219393 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:34.219436 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:08:34.219507 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:34.219842 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:34.219851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:34.219856 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:08:34.220036 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 02:08:34.220316 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260808 02:08:34.222489 19671 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34771
I20260808 02:08:34.222690 19671 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:34.223073 19671 heartbeater.cc:511] Master 127.5.209.126:34771 requested a full tablet report, sending...
I20260808 02:08:34.224030 19018 ts_manager.cc:194] Registered new tserver with Master: 5834f1d16c4e48e3a360c484514b085f (127.5.209.67:39647)
I20260808 02:08:34.233738 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:08:34.236569 19076 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:08:34.235717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.5.209.126:34771)
0808 02:08:34.235850 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:34.235853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:34.236005 (+ 152us) server_negotiation.cc:407] Connection header received
0808 02:08:34.236102 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:34.236105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:34.236137 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:34.236203 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:34.236448 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50256: BlockingRecv error: recv got EOF from 127.0.0.1:50256 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:08:34.236919 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 19106
I20260808 02:08:34.251813 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 19288
I20260808 02:08:34.261739 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 19484
I20260808 02:08:34.276860 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 18983
2026-08-08T02:08:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (869 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:34 dist-test-slave-bw88 krb5kdc[19686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:34 dist-test-slave-bw88 krb5kdc[19686](info): set up 2 sockets
Aug 08 02:08:34 dist-test-slave-bw88 krb5kdc[19686](info): commencing operation
krb5kdc: starting...
W20260808 02:08:35.513542 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:35 dist-test-slave-bw88 krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:08:35.640436 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33435
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:35.756814 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:35.757073 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:35.757121 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:35.761034 19708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:08:35.761098 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:35.761118 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:35.761134 19708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:08:35.761166 19708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:08:35.766388 19708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:35.767352 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:35.768339 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:35.777117 19718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:35.777117 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:35.777164 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:35.777367 19708 server_base.cc:1034] running on GCE node
I20260808 02:08:35.778065 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:35.779438 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:35.780629 19708 hybrid_clock.cc:648] HybridClock initialized: now 1786154915780605 us; error 46 us; skew 500 ppm
Aug 08 02:08:35 dist-test-slave-bw88 krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154915, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:35.783433 19708 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:08:35.784358 19708 webserver.cc:460] Webserver started at http://127.5.209.126:45843/ using document root <none> and password file <none>
I20260808 02:08:35.784932 19708 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:35.784994 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:35.785236 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:35.787374 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "55a4773e96b7425881d3b9ccc7443586"
format_stamp: "Formatted at 2026-08-08 02:08:35 on dist-test-slave-bw88"
server_key: "5c8e960aa6a9bf9d950941a5b0e16fc3"
server_key_iv: "72c2f2e1418b42193919217ed6ac689d"
server_key_version: "encryptionkey@0"
I20260808 02:08:35.787962 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "55a4773e96b7425881d3b9ccc7443586"
format_stamp: "Formatted at 2026-08-08 02:08:35 on dist-test-slave-bw88"
server_key: "5c8e960aa6a9bf9d950941a5b0e16fc3"
server_key_iv: "72c2f2e1418b42193919217ed6ac689d"
server_key_version: "encryptionkey@0"
I20260808 02:08:35.791651 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 02:08:35.793984 19725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:35.795135 19708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:08:35.795290 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "55a4773e96b7425881d3b9ccc7443586"
format_stamp: "Formatted at 2026-08-08 02:08:35 on dist-test-slave-bw88"
server_key: "5c8e960aa6a9bf9d950941a5b0e16fc3"
server_key_iv: "72c2f2e1418b42193919217ed6ac689d"
server_key_version: "encryptionkey@0"
I20260808 02:08:35.795383 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:35.839386 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:35.842461 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:35.842690 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:35.850303 19708 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33435
I20260808 02:08:35.850595 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33435 every 8 connection(s)
I20260808 02:08:35.851346 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:08:35.854241 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:35.857622 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 19708
I20260808 02:08:35.857769 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:08:35.858050 5957 external_mini_cluster.cc:1428] Setting key 76a4bc208c8395b7bf236b8f9acb45e9
I20260808 02:08:35.860123 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: Bootstrap starting.
I20260808 02:08:35.862759 19801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:35.863619 19801 log.cc:824] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:35.866074 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: No bootstrap required, opened a new log
Aug 08 02:08:35 dist-test-slave-bw88 krb5kdc[19686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:35.870811 19801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } }
I20260808 02:08:35.871096 19801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:35.871173 19801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a4773e96b7425881d3b9ccc7443586, State: Initialized, Role: FOLLOWER
I20260808 02:08:35.871703 19801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } }
I20260808 02:08:35.871868 19801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:08:35.871971 19801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:08:35.872090 19801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:35.873106 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:35.859434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.5.209.126:33435)
0808 02:08:35.859849 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:35.859879 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:35.859916 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:08:35.860574 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:35.860601 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:35.860924 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:35.861210 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:35.862143 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:35.863222 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:35.864341 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:35.864941 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:35.865064 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:35.867777 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:35.867784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:35.867802 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:35.867835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:35.869846 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:35.870307 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:35.870313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:35.870319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:35.870457 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:35.870841 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:35.870845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:35.870846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:35.871345 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:35.871553 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:35.871834 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 02:08:35.872139 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
I20260808 02:08:35.873260 19801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } }
I20260808 02:08:35.873632 19801 leader_election.cc:302] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55a4773e96b7425881d3b9ccc7443586; no voters:
I20260808 02:08:35.873955 19801 leader_election.cc:288] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:08:35.874079 19806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:35.874346 19806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [term 1 LEADER]: Becoming Leader. State: Replica: 55a4773e96b7425881d3b9ccc7443586, State: Running, Role: LEADER
I20260808 02:08:35.874687 19806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } }
I20260808 02:08:35.876402 19801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:08:35.876461 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55a4773e96b7425881d3b9ccc7443586. Latest consensus state: current_term: 1 leader_uuid: "55a4773e96b7425881d3b9ccc7443586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } } }
I20260808 02:08:35.876572 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:35.876525 19807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55a4773e96b7425881d3b9ccc7443586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55a4773e96b7425881d3b9ccc7443586" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33435 } } }
I20260808 02:08:35.876657 19807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586 [sys.catalog]: This master's current role is: LEADER
I20260808 02:08:35.877588 19810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:08:35.880379 19810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:08:35.885895 19810 catalog_manager.cc:1347] Generated new cluster ID: 18a88df45ce541ed816fbd54a475564d
I20260808 02:08:35.885960 19810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:08:35.893262 19810 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:08:35.894153 19810 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:08:35.903314 19810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: Generated new TSK 0
I20260808 02:08:35.904034 19810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:08:35.934482 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:36.050109 19833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:36.050346 19833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:36.050387 19833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:36.054297 19833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:36.054347 19833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:36.054400 19833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:08:36.059676 19833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:36.060690 19833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:36.061753 19833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:36.071645 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:36.071628 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:36.071652 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:36.071879 19833 server_base.cc:1034] running on GCE node
I20260808 02:08:36.072320 19833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:36.073050 19833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:36.074218 19833 hybrid_clock.cc:648] HybridClock initialized: now 1786154916074175 us; error 56 us; skew 500 ppm
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:36.077209 19833 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:08:36.078176 19833 webserver.cc:460] Webserver started at http://127.5.209.65:40093/ using document root <none> and password file <none>
I20260808 02:08:36.078794 19833 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:36.078896 19833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:36.079077 19833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:36.081015 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "f8cde1d81ce14b0c896e0b3fcb155af3"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "a3f27d1042f6f8cd1370e452e8dd25dc"
server_key_iv: "99b791d48b34e60747b5aed691c5af0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.081615 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "f8cde1d81ce14b0c896e0b3fcb155af3"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "a3f27d1042f6f8cd1370e452e8dd25dc"
server_key_iv: "99b791d48b34e60747b5aed691c5af0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.085220 19833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:08:36.087582 19857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.088768 19833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:36.088892 19833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "f8cde1d81ce14b0c896e0b3fcb155af3"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "a3f27d1042f6f8cd1370e452e8dd25dc"
server_key_iv: "99b791d48b34e60747b5aed691c5af0d"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.088995 19833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:36.115358 19833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:36.118796 19833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:36.119017 19833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:36.119647 19833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:36.120515 19833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:36.120563 19833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.120644 19833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:36.120676 19833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.129500 19833 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:43915
I20260808 02:08:36.129540 20010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:43915 every 8 connection(s)
I20260808 02:08:36.130402 19833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:08:36.131174 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 19833
I20260808 02:08:36.131342 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:08:36.131647 5957 external_mini_cluster.cc:1428] Setting key 89d8573a68dcd2e7395ace78c2f70ff6
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:08:36.145052 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.132440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:40501 (local address 127.5.209.126:33435)
0808 02:08:36.132554 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.132557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.133695 (+ 1138us) server_negotiation.cc:407] Connection header received
0808 02:08:36.135089 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.135093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.135160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.135238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:36.136407 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.137093 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.138375 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.138633 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.138755 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.141268 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:36.141272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:36.141275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:36.141310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:36.142899 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.143435 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.143439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.143441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.143536 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.143900 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.143903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.143905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.144110 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:36.144223 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.144736 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.144867 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 02:08:36.145902 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.132963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33435 (local address 127.5.209.65:40501)
0808 02:08:36.133533 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.133575 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.134876 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.135409 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.135416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.135776 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:36.136252 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.136266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.137255 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.137258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.138234 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.138246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.138803 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.138807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.138984 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.139611 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:36.139628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:36.141076 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:36.143028 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.143033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.143048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.143341 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.143641 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.143645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.143647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.143794 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.144241 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:36.144248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:36.144493 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.144692 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.145016 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:08:36.147341 20011 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33435
I20260808 02:08:36.147606 20011 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:36.148105 20011 heartbeater.cc:511] Master 127.5.209.126:33435 requested a full tablet report, sending...
I20260808 02:08:36.149763 19754 ts_manager.cc:194] Registered new tserver with Master: f8cde1d81ce14b0c896e0b3fcb155af3 (127.5.209.65:43915)
I20260808 02:08:36.150876 19754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:40501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:08:36.164546 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:36.284358 20017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:36.284641 20017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:36.284686 20017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:36.288640 20017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:36.288692 20017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:36.288749 20017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:08:36.294490 20017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:36.295543 20017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:36.296561 20017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:36.306921 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:36.306946 20032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:36.306946 20034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:36.307134 20017 server_base.cc:1034] running on GCE node
I20260808 02:08:36.307626 20017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:36.308310 20017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:36.309477 20017 hybrid_clock.cc:648] HybridClock initialized: now 1786154916309462 us; error 24 us; skew 500 ppm
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:36.312301 20017 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:08:36.313241 20017 webserver.cc:460] Webserver started at http://127.5.209.66:33009/ using document root <none> and password file <none>
I20260808 02:08:36.313865 20017 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:36.313930 20017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:36.314119 20017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:36.316229 20017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "1c18b63b4698462483bfee694631551a"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "b1bce0477fd676a66ee9d62354b2299f"
server_key_iv: "85df1173b9d77ed10b0b52df902774da"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.316764 20017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "1c18b63b4698462483bfee694631551a"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "b1bce0477fd676a66ee9d62354b2299f"
server_key_iv: "85df1173b9d77ed10b0b52df902774da"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.320600 20017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:08:36.323105 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.324472 20017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:36.324626 20017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "1c18b63b4698462483bfee694631551a"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "b1bce0477fd676a66ee9d62354b2299f"
server_key_iv: "85df1173b9d77ed10b0b52df902774da"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.324746 20017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:36.343924 20017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:36.347219 20017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:36.347448 20017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:36.348097 20017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:36.348999 20017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:36.349076 20017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.349159 20017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:36.349211 20017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.358436 20017 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:41159
I20260808 02:08:36.358466 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:41159 every 8 connection(s)
I20260808 02:08:36.359369 20017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:08:36.362160 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 20017
I20260808 02:08:36.362293 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:08:36.362632 5957 external_mini_cluster.cc:1428] Setting key 9b96ca6d55fc5c8c44c3fc097e9803b5
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:08:36.374593 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.361213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:45791 (local address 127.5.209.126:33435)
0808 02:08:36.361409 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.361414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.362259 (+ 845us) server_negotiation.cc:407] Connection header received
0808 02:08:36.363848 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.363852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.363908 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.363990 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:36.365210 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.365833 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.367640 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.367865 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.367947 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.370925 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:36.370929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:36.370931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:36.370954 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:36.372412 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.372932 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.372936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.372937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.373022 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.373423 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.373426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.373428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.373624 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:36.373732 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.374282 (+ 550us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.374412 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:08:36.375542 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.361633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33435 (local address 127.5.209.66:45791)
0808 02:08:36.362105 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.362149 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.363631 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.364170 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.364177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.364530 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:36.365046 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.365059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.365972 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.365980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.367477 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.367495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.368032 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.368039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.368296 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.368947 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:36.368973 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:36.370751 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:36.372539 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.372545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.372560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.372823 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.373125 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.373128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.373130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.373312 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.373747 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:36.373756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:36.374027 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.374220 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.374556 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260808 02:08:36.376775 20210 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:08:36.377081 20210 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:36.377750 20210 heartbeater.cc:511] Master 127.5.209.126:33435 requested a full tablet report, sending...
I20260808 02:08:36.378990 19754 ts_manager.cc:194] Registered new tserver with Master: 1c18b63b4698462483bfee694631551a (127.5.209.66:41159)
I20260808 02:08:36.379647 19754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:45791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:08:36.392843 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:08:36.509856 20219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:08:36.510103 20219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:08:36.510180 20219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:08:36.514166 20219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:08:36.514222 20219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:08:36.514322 20219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:08:36.519490 20219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:08:36.520505 20219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:08:36.521622 20219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:08:36.531523 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:08:36.531539 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:36.531651 20219 server_base.cc:1034] running on GCE node
W20260808 02:08:36.531522 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:08:36.532150 20219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:08:36.532797 20219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:08:36.533964 20219 hybrid_clock.cc:648] HybridClock initialized: now 1786154916533928 us; error 47 us; skew 500 ppm
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:08:36.536752 20219 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:08:36.537627 20219 webserver.cc:460] Webserver started at http://127.5.209.67:34397/ using document root <none> and password file <none>
I20260808 02:08:36.538159 20219 fs_manager.cc:359] Metadata directory not provided
I20260808 02:08:36.538239 20219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:08:36.538401 20219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:08:36.540377 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "87eb28fc102a428ba1e40af544ec63b1"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "13ced49a0c899749f2132792237e2267"
server_key_iv: "d046f47cd0d6d9f9d0b11aa0c4dde8f9"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.540879 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "87eb28fc102a428ba1e40af544ec63b1"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "13ced49a0c899749f2132792237e2267"
server_key_iv: "d046f47cd0d6d9f9d0b11aa0c4dde8f9"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.544528 20219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:08:36.547102 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.548381 20219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:08:36.548580 20219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "87eb28fc102a428ba1e40af544ec63b1"
format_stamp: "Formatted at 2026-08-08 02:08:36 on dist-test-slave-bw88"
server_key: "13ced49a0c899749f2132792237e2267"
server_key_iv: "d046f47cd0d6d9f9d0b11aa0c4dde8f9"
server_key_version: "encryptionkey@0"
I20260808 02:08:36.548702 20219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:08:36.573920 20219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:08:36.578022 20219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:08:36.578249 20219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:08:36.578920 20219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:08:36.579913 20219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:08:36.579988 20219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.580065 20219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:08:36.580134 20219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:08:36.589785 20219 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:45853
I20260808 02:08:36.589854 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:45853 every 8 connection(s)
I20260808 02:08:36.590780 20219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:08:36.600121 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 20219
I20260808 02:08:36.600255 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:08:36.600564 5957 external_mini_cluster.cc:1428] Setting key 39e4feb026a3bd63d8390db80954084d
Aug 08 02:08:36 dist-test-slave-bw88 krb5kdc[19686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154916, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:08:36.605479 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.592821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:57895 (local address 127.5.209.126:33435)
0808 02:08:36.592974 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.592978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.593930 (+ 952us) server_negotiation.cc:407] Connection header received
0808 02:08:36.595373 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.595377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.595434 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.595540 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:36.597002 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.597767 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.599000 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.599228 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.599311 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.602114 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:36.602128 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:36.602131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:36.602158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:36.603641 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.604110 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.604112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.604113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.604190 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.604484 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.604486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.604487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.604672 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:36.604749 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.605186 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.605282 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:08:36.606316 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.593228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33435 (local address 127.5.209.67:57895)
0808 02:08:36.593784 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.593828 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.595163 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.595724 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.595731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.596088 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:08:36.596824 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.596846 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.597903 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.597906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.598895 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.598905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.599362 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.599364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.599509 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.600167 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:08:36.600208 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:08:36.601941 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:08:36.603749 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.603754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.603765 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.604016 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.604272 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:08:36.604274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:08:36.604275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:08:36.604403 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:08:36.604764 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:08:36.604769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:08:36.604982 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.605135 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.605436 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":170,"threads_started":1}
I20260808 02:08:36.607471 20395 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33435
I20260808 02:08:36.607698 20395 heartbeater.cc:464] Registering TS with master...
I20260808 02:08:36.608140 20395 heartbeater.cc:511] Master 127.5.209.126:33435 requested a full tablet report, sending...
I20260808 02:08:36.609136 19754 ts_manager.cc:194] Registered new tserver with Master: 87eb28fc102a428ba1e40af544ec63b1 (127.5.209.67:45853)
I20260808 02:08:36.609724 19754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:57895
I20260808 02:08:36.614485 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:08:36.623843 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.616346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.5.209.126:33435)
0808 02:08:36.616506 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.616512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.616713 (+ 201us) server_negotiation.cc:407] Connection header received
0808 02:08:36.616866 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.616869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.616914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.617011 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:08:36.617805 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.618315 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.619186 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.619413 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.619521 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.620789 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:08:36.620793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:08:36.620796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:08:36.620821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:08:36.622124 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.622560 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.622564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.622566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.622653 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:08:36.622943 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:08:36.622947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:08:36.622949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:08:36.623180 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:08:36.623326 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.623565 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.623677 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:08:36.626518 19754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:08:36.629716 19754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:08:36.644443 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.638325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.5.209.67:45853)
0808 02:08:36.638871 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.638884 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.639231 (+ 347us) server_negotiation.cc:407] Connection header received
0808 02:08:36.639318 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.639323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.639528 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.639817 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:36.640229 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641095 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.643112 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.643681 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.643832 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.643996 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.644179 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.644276 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":122,"threads_started":1}
I20260808 02:08:36.644419 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.638062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:45853 (local address 127.0.0.1:34584)
0808 02:08:36.638904 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.638921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.639061 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.640001 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.640003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.640018 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:36.640094 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.640106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641255 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.641260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.642964 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.642977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.643833 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.643837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.643947 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.643986 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.644141 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.644267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":69,"spinlock_wait_cycles":5760,"thread_start_us":109,"threads_started":1}
I20260808 02:08:36.644598 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.637966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41159 (local address 127.0.0.1:55766)
0808 02:08:36.638846 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.638900 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.639061 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.639655 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.639662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.639687 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:36.639827 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.639834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641517 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.641520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.642998 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.643008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.644118 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.644122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.644218 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.644240 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.644293 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.644369 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":189,"spinlock_wait_cycles":4736,"thread_start_us":86,"threads_started":1}
I20260808 02:08:36.644974 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.638848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:43915 (local address 127.0.0.1:33260)
0808 02:08:36.639503 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:36.639519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:36.639602 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:36.640320 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:36.640323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:36.640343 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:36.640428 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.640434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641783 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.641787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.642984 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:36.642994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.644121 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.644123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:36.644433 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.644465 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:08:36.644753 (+ 288us) client_negotiation.cc:240] Negotiation successful
0808 02:08:36.644840 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":291,"thread_start_us":97,"threads_started":1}
I20260808 02:08:36.645467 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.638092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.5.209.66:41159)
0808 02:08:36.639099 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.639112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.639139 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:08:36.639226 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.639231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.639450 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.639990 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:36.640155 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641385 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.643131 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.643968 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.644986 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.645157 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.645211 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.645302 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":141,"threads_started":1}
I20260808 02:08:36.645488 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:36.639372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.5.209.65:43915)
0808 02:08:36.639853 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:36.639871 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:36.639896 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:08:36.639976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:36.639980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:36.640170 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:36.640328 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:36.640555 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.641651 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.643149 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:36.643968 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:36.644168 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:36.644312 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:36.645244 (+ 932us) server_negotiation.cc:299] Negotiation successful
0808 02:08:36.645331 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":96,"threads_started":1}
I20260808 02:08:36.647737 20308 tablet_service.cc:1467] Processing CreateTablet for tablet 7cebf5b248684b6e8513a6f659f7975c (DEFAULT_TABLE table=test-table [id=51c08cbadbb24a8a998898272212a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:36.647769 20116 tablet_service.cc:1467] Processing CreateTablet for tablet 7cebf5b248684b6e8513a6f659f7975c (DEFAULT_TABLE table=test-table [id=51c08cbadbb24a8a998898272212a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:36.648741 20116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cebf5b248684b6e8513a6f659f7975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:36.648752 20308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cebf5b248684b6e8513a6f659f7975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:36.654426 19924 tablet_service.cc:1467] Processing CreateTablet for tablet 7cebf5b248684b6e8513a6f659f7975c (DEFAULT_TABLE table=test-table [id=51c08cbadbb24a8a998898272212a453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:08:36.655489 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cebf5b248684b6e8513a6f659f7975c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:08:36.655838 20419 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Bootstrap starting.
I20260808 02:08:36.656564 20422 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Bootstrap starting.
I20260808 02:08:36.658413 20419 tablet_bootstrap.cc:654] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:36.659165 20419 log.cc:824] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:36.659271 20422 tablet_bootstrap.cc:654] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:36.660084 20422 log.cc:824] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:36.661643 20419 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: No bootstrap required, opened a new log
I20260808 02:08:36.661852 20419 ts_tablet_manager.cc:1397] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:08:36.662479 20422 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: No bootstrap required, opened a new log
I20260808 02:08:36.662685 20422 ts_tablet_manager.cc:1397] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 02:08:36.662796 20427 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Bootstrap starting.
I20260808 02:08:36.664772 20427 tablet_bootstrap.cc:654] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Neither blocks nor log segments found. Creating new log.
I20260808 02:08:36.665509 20427 log.cc:824] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:08:36.665922 20419 raft_consensus.cc:348] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.666213 20419 raft_consensus.cc:374] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:36.666278 20419 raft_consensus.cc:729] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87eb28fc102a428ba1e40af544ec63b1, State: Initialized, Role: FOLLOWER
I20260808 02:08:36.666738 20419 consensus_queue.cc:260] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.666739 20422 raft_consensus.cc:348] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.666985 20422 raft_consensus.cc:374] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:36.667062 20422 raft_consensus.cc:729] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c18b63b4698462483bfee694631551a, State: Initialized, Role: FOLLOWER
I20260808 02:08:36.667495 20422 consensus_queue.cc:260] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.667862 20427 tablet_bootstrap.cc:492] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: No bootstrap required, opened a new log
I20260808 02:08:36.667863 20419 ts_tablet_manager.cc:1428] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:08:36.668052 20427 ts_tablet_manager.cc:1397] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 02:08:36.668231 20395 heartbeater.cc:503] Master 127.5.209.126:33435 was elected leader, sending a full tablet report...
I20260808 02:08:36.668308 20210 heartbeater.cc:503] Master 127.5.209.126:33435 was elected leader, sending a full tablet report...
I20260808 02:08:36.668476 20422 ts_tablet_manager.cc:1428] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:08:36.673844 20427 raft_consensus.cc:348] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.674151 20427 raft_consensus.cc:374] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:08:36.674223 20427 raft_consensus.cc:729] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cde1d81ce14b0c896e0b3fcb155af3, State: Initialized, Role: FOLLOWER
I20260808 02:08:36.674589 20427 consensus_queue.cc:260] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:36.675271 20011 heartbeater.cc:503] Master 127.5.209.126:33435 was elected leader, sending a full tablet report...
I20260808 02:08:36.675633 20427 ts_tablet_manager.cc:1428] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260808 02:08:36.841840 20396 tablet.cc:2369] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:08:36.860531 20212 tablet.cc:2369] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:08:36.881762 20012 tablet.cc:2369] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:08:37.016919 20429 raft_consensus.cc:482] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:08:37.017160 20429 raft_consensus.cc:504] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:37.018412 20429 leader_election.cc:288] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c18b63b4698462483bfee694631551a (127.5.209.66:41159), f8cde1d81ce14b0c896e0b3fcb155af3 (127.5.209.65:43915)
I20260808 02:08:37.022465 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.018704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:43915 (local address 127.5.209.67:42183)
0808 02:08:37.018820 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:37.018837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:37.018947 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:37.019272 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:37.019275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:37.019319 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:37.019462 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:37.019469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.020378 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.020382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:37.021299 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:37.021310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.022002 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.022007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:37.022134 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:37.022164 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:08:37.022234 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:08:37.022317 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":49}
I20260808 02:08:37.022500 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.018847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:42183 (local address 127.5.209.65:43915)
0808 02:08:37.018979 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:37.018983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:37.019001 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:08:37.019073 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:37.019076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:37.019144 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:37.019253 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:37.019639 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.020263 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.021443 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.021873 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.021972 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:37.022044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:37.022284 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:08:37.022366 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:08:37.022557 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.018847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:38221 (local address 127.5.209.66:41159)
0808 02:08:37.018978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:37.018984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:37.019344 (+ 360us) server_negotiation.cc:407] Connection header received
0808 02:08:37.019458 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:37.019462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:37.019527 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:37.019626 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:08:37.019899 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.020418 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.021538 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.021963 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.022080 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:37.022131 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:37.022352 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:08:37.022430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:08:37.022733 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.018704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41159 (local address 127.5.209.67:38221)
0808 02:08:37.019225 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:37.019242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:08:37.019344 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:08:37.019644 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:08:37.019647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:08:37.019666 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:08:37.019779 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:37.019786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.020535 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.020538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:37.021396 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:08:37.021410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.022109 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.022112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:08:37.022192 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:37.022211 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:08:37.022313 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:08:37.022381 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":127,"thread_start_us":121,"threads_started":1}
I20260808 02:08:37.023205 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cebf5b248684b6e8513a6f659f7975c" candidate_uuid: "87eb28fc102a428ba1e40af544ec63b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" is_pre_election: true
I20260808 02:08:37.023437 20148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cebf5b248684b6e8513a6f659f7975c" candidate_uuid: "87eb28fc102a428ba1e40af544ec63b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c18b63b4698462483bfee694631551a" is_pre_election: true
I20260808 02:08:37.023530 19947 raft_consensus.cc:2454] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87eb28fc102a428ba1e40af544ec63b1 in term 0.
I20260808 02:08:37.023761 20148 raft_consensus.cc:2454] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87eb28fc102a428ba1e40af544ec63b1 in term 0.
I20260808 02:08:37.024071 20248 leader_election.cc:302] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87eb28fc102a428ba1e40af544ec63b1, f8cde1d81ce14b0c896e0b3fcb155af3; no voters:
I20260808 02:08:37.024369 20429 raft_consensus.cc:2781] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:08:37.024451 20429 raft_consensus.cc:482] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:08:37.024482 20429 raft_consensus.cc:3037] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:37.025633 20429 raft_consensus.cc:504] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:37.026027 20429 leader_election.cc:288] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1c18b63b4698462483bfee694631551a (127.5.209.66:41159), f8cde1d81ce14b0c896e0b3fcb155af3 (127.5.209.65:43915)
I20260808 02:08:37.026399 20148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cebf5b248684b6e8513a6f659f7975c" candidate_uuid: "87eb28fc102a428ba1e40af544ec63b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c18b63b4698462483bfee694631551a"
I20260808 02:08:37.026408 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cebf5b248684b6e8513a6f659f7975c" candidate_uuid: "87eb28fc102a428ba1e40af544ec63b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3"
I20260808 02:08:37.026507 20148 raft_consensus.cc:3037] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:37.026546 19947 raft_consensus.cc:3037] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:08:37.027624 20148 raft_consensus.cc:2454] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87eb28fc102a428ba1e40af544ec63b1 in term 1.
I20260808 02:08:37.027706 19947 raft_consensus.cc:2454] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87eb28fc102a428ba1e40af544ec63b1 in term 1.
I20260808 02:08:37.027962 20241 leader_election.cc:302] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c18b63b4698462483bfee694631551a, 87eb28fc102a428ba1e40af544ec63b1; no voters:
I20260808 02:08:37.028154 20429 raft_consensus.cc:2781] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:08:37.028396 20429 raft_consensus.cc:686] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 1 LEADER]: Becoming Leader. State: Replica: 87eb28fc102a428ba1e40af544ec63b1, State: Running, Role: LEADER
I20260808 02:08:37.028726 20429 consensus_queue.cc:237] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } }
I20260808 02:08:37.031960 19754 catalog_manager.cc:5495] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87eb28fc102a428ba1e40af544ec63b1 (127.5.209.67). New cstate: current_term: 1 leader_uuid: "87eb28fc102a428ba1e40af544ec63b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87eb28fc102a428ba1e40af544ec63b1" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 } health_report { overall_health: UNKNOWN } } }
I20260808 02:08:37.096769 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.091772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.5.209.67:45853)
0808 02:08:37.091936 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 02:08:37.091943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:08:37.091959 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:08:37.092085 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:08:37.092089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:08:37.092164 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:08:37.092303 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:08:37.092609 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.093475 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.094905 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:08:37.095347 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:08:37.095457 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:08:37.095872 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:08:37.096332 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:08:37.096433 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:08:37.096518 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:08:37.096574 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:08:37.103472 20148 raft_consensus.cc:1261] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 1 FOLLOWER]: Refusing update from remote peer 87eb28fc102a428ba1e40af544ec63b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:37.103598 19947 raft_consensus.cc:1261] T 7cebf5b248684b6e8513a6f659f7975c P f8cde1d81ce14b0c896e0b3fcb155af3 [term 1 FOLLOWER]: Refusing update from remote peer 87eb28fc102a428ba1e40af544ec63b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:08:37.104348 20434 consensus_queue.cc:1035] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c18b63b4698462483bfee694631551a" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:37.104575 20429 consensus_queue.cc:1035] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cde1d81ce14b0c896e0b3fcb155af3" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:08:37.111986 20439 mvcc.cc:205] Tried to move back new op lower bound from 7316090540447481856 to 7316090540152324096. Current Snapshot: MvccSnapshot[applied={T|T < 7316090540447481856}]
I20260808 02:08:37.121738 19754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:37.121899 19754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:08:37.124567 19754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55a4773e96b7425881d3b9ccc7443586: Sending DeleteTablet for 3 replicas of tablet 7cebf5b248684b6e8513a6f659f7975c
I20260808 02:08:37.125509 20116 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cebf5b248684b6e8513a6f659f7975c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:37 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:55766
I20260808 02:08:37.125633 20308 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cebf5b248684b6e8513a6f659f7975c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:37 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34584
I20260808 02:08:37.125627 19924 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cebf5b248684b6e8513a6f659f7975c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:08:37 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33260
I20260808 02:08:37.126175 20452 tablet_replica.cc:330] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: stopping tablet replica
I20260808 02:08:37.126508 20452 raft_consensus.cc:2229] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:08:37.126884 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 19833
I20260808 02:08:37.126886 20452 raft_consensus.cc:2258] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:37.137660 20455 tablet_replica.cc:330] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: stopping tablet replica
I20260808 02:08:37.138262 20455 raft_consensus.cc:2229] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:08:37.138908 20455 raft_consensus.cc:2258] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:08:37.141477 20455 ts_tablet_manager.cc:1905] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:37.145362 20455 ts_tablet_manager.cc:1918] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:37.145509 20455 log.cc:1196] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/7cebf5b248684b6e8513a6f659f7975c
I20260808 02:08:37.145913 20455 ts_tablet_manager.cc:1939] T 7cebf5b248684b6e8513a6f659f7975c P 87eb28fc102a428ba1e40af544ec63b1: Deleting consensus metadata
I20260808 02:08:37.147519 19729 catalog_manager.cc:4841] TS 87eb28fc102a428ba1e40af544ec63b1 (127.5.209.67:45853): tablet 7cebf5b248684b6e8513a6f659f7975c (table test-table [id=51c08cbadbb24a8a998898272212a453]) successfully deleted
I20260808 02:08:37.148519 20452 ts_tablet_manager.cc:1905] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:08:37.151016 20452 ts_tablet_manager.cc:1918] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:08:37.151153 20452 log.cc:1196] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/7cebf5b248684b6e8513a6f659f7975c
I20260808 02:08:37.151449 20452 ts_tablet_manager.cc:1939] T 7cebf5b248684b6e8513a6f659f7975c P 1c18b63b4698462483bfee694631551a: Deleting consensus metadata
I20260808 02:08:37.152768 19726 catalog_manager.cc:4841] TS 1c18b63b4698462483bfee694631551a (127.5.209.66:41159): tablet 7cebf5b248684b6e8513a6f659f7975c (table test-table [id=51c08cbadbb24a8a998898272212a453]) successfully deleted
W20260808 02:08:37.155263 19729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:43915 (error 108)
I20260808 02:08:37.156137 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 20017
I20260808 02:08:37.156437 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:08:37.155993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:43915 (local address 127.0.0.1:33272)
0808 02:08:37.156224 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:08:37.156310 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:43915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
W20260808 02:08:37.156921 19729 catalog_manager.cc:4584] TS f8cde1d81ce14b0c896e0b3fcb155af3 (127.5.209.65:43915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cebf5b248684b6e8513a6f659f7975c: Network error: Client connection negotiation failed: client connection to 127.5.209.65:43915: connect: Connection refused (error 111)
I20260808 02:08:37.167157 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 20219
I20260808 02:08:37.178721 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 19708
2026-08-08T02:08:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2903 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:08:37 dist-test-slave-bw88 krb5kdc[20460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:08:37 dist-test-slave-bw88 krb5kdc[20460](info): set up 2 sockets
Aug 08 02:08:37 dist-test-slave-bw88 krb5kdc[20460](info): commencing operation
krb5kdc: starting...
W20260808 02:08:38.375484 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:08:38 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:08:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangeradmin_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangeradmin_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangerlookup_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangerlookup_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
I20260808 02:08:38.552338 5957 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskguwdHL/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/postgres -l logfile start
2026-08-08 02:08:41.127 UTC [20499] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:08:41.127 UTC [20499] LOG: listening on IPv4 address "127.5.209.84", port 52237
2026-08-08 02:08:41.131 UTC [20499] LOG: listening on Unix socket "/tmp/.s.PGSQL.52237"
2026-08-08 02:08:41.168 UTC [20502] LOG: database system was shut down at 2026-08-08 02:08:39 UTC
2026-08-08 02:08:41.172 UTC [20499] LOG: database system is ready to accept connections
I20260808 02:08:42.329089 5957 mini_postgres.cc:96] Postgres bound to 52237
2026-08-08 02:08:42.337 UTC [20517] FATAL: database "slave" does not exist
127.5.209.84:52237 - accepting connections
I20260808 02:08:42.338241 5957 mini_ranger.cc:161] Starting Ranger...
I20260808 02:08:42.354543 5957 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:08:42.858995 5957 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:08:42.859117 5957 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin
2026-08-08 02:08:43,147 [I] DB FLAVOR :POSTGRES
2026-08-08 02:08:43,148 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:08:43,148 [I] Checking connection..
2026-08-08 02:08:43,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:08:43,412 [I] Checking connection passed.
2026-08-08 02:08:43,412 [I] --------- Verifying version history table ---------
2026-08-08 02:08:43,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:08:43,680 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:08:43,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:08:43,950 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:08:43,951 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:08:43,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:08:44.215 UTC [20608] WARNING: there is no transaction in progress
2026-08-08 02:08:44,223 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:08:44,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:08:44,468 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:08:44,469 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:08:44,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:08:44,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:08:44,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bw88.c.gcp-upstream.internal','N') ;"
2026-08-08 02:08:45,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:08:45,561 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:08:45,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:08:45,829 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:08:45,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:08:46,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:08:46,319 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:08:46,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:08:47.292 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.309 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.325 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.340 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.356 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.373 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.427 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.434 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.448 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.458 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.469 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.482 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.491 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.498 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.505 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:47.512 UTC [20757] WARNING: there is no transaction in progress
2026-08-08 02:08:48,155 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:08:48,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:08:48,421 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:08:48,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:08:48,671 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:08:48,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:08:48,942 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:08:48,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:08:49,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:08:49,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bw88.c.gcp-upstream.internal';"
2026-08-08 02:08:49,689 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:08:49,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:08:49,939 [I] DB_PATCHES have already been applied
I20260808 02:08:49.945895 5957 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskguwdHL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:08:49.945987 5957 mini_ranger.cc:193] Using host: 127.5.209.84
I20260808 02:08:49.950655 5957 mini_ranger.cc:227] Ranger admin URL: http://127.5.209.84:44521
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:08:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:08:51.231278 5957 mini_ranger.cc:160] Time spent starting Ranger: real 8.893s user 0.004s sys 0.008s
Aug 08, 2026 2:09:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:09:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 2:09:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:09:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:09:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:09:13.985327 5957 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangerkms_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/rangerkms_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.209.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.209.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:09:14 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:09:14.080008 5957 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:09:14.090505 5957 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:09:14.969288 5957 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:09:14.969425 5957 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms
I20260808 02:09:14.969545 5957 mini_ranger_kms.cc:227] Using postgres at 127.5.209.84:52237
2026-08-08 02:09:15,681 [I] DB FLAVOR :POSTGRES
2026-08-08 02:09:15,681 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:09:15,681 [I] Checking connection
2026-08-08 02:09:15,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:09:15,956 [I] connection success
2026-08-08 02:09:15,957 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:09:15,957 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:09:15,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:09:16,201 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:09:16,201 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:09:16,201 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:09:16,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:52237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:09:16,474 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:09:16.995848 5957 mini_ranger_kms.cc:310] Created kms service
I20260808 02:09:17.204777 5957 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:09:17.263768 5957 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:09:17 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:17.711270 5957 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:09:17.711434 5957 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms:/tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:09:17.711454 5957 mini_ranger_kms.cc:240] Using host: 127.5.209.84
I20260808 02:09:17.716651 5957 mini_ranger_kms.cc:276] Ranger KMS PID: 21018
I20260808 02:09:17.716797 5957 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.209.84:43931
Aug 08, 2026 2:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:09:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:09:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:09:19 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786154959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 02:09:20 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:20.348795 5957 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.269s user 0.010s sys 0.274s
I20260808 02:09:20.348922 5957 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:09:20.349038 5957 mini_ranger_kms.cc:401] 127.5.209.84:43931/kms/v1/keys
Aug 08 02:09:20 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
Aug 08 02:09:23 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:09:23.036927 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38825
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:38825
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:43931/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:09:23.163236 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:09:23.163527 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:09:23.163566 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:09:23.167708 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:09:23.167778 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:09:23.167798 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:09:23.167815 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:09:23.167831 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:09:23.173641 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.209.84:43931/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:38825
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:09:23.174912 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:09:23.176041 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:09:23.185119 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:23.185292 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:23.185372 21091 server_base.cc:1034] running on GCE node
W20260808 02:09:23.185191 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:23.186005 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:09:23.187202 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:09:23.188485 21091 hybrid_clock.cc:648] HybridClock initialized: now 1786154963188458 us; error 44 us; skew 500 ppm
Aug 08 02:09:23 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:09:23.195936 21091 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:09:23.197113 21091 webserver.cc:460] Webserver started at http://127.5.209.126:41125/ using document root <none> and password file <none>
I20260808 02:09:23.197831 21091 fs_manager.cc:359] Metadata directory not provided
I20260808 02:09:23.197911 21091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:09:23.198153 21091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:09:23 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154963, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:23.249084 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "5c1b78857815455aa0f69cf175dae1f7"
format_stamp: "Formatted at 2026-08-08 02:09:23 on dist-test-slave-bw88"
server_key: "f172bc626509c65811fa51b52fb93854"
server_key_iv: "4a9326b9cf0978cfe5011845fb43ee63"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:23.249861 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "5c1b78857815455aa0f69cf175dae1f7"
format_stamp: "Formatted at 2026-08-08 02:09:23 on dist-test-slave-bw88"
server_key: "f172bc626509c65811fa51b52fb93854"
server_key_iv: "4a9326b9cf0978cfe5011845fb43ee63"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:23.254124 21091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:09:23.268033 21107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:23.269634 21091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:09:23.269773 21091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "5c1b78857815455aa0f69cf175dae1f7"
format_stamp: "Formatted at 2026-08-08 02:09:23 on dist-test-slave-bw88"
server_key: "f172bc626509c65811fa51b52fb93854"
server_key_iv: "4a9326b9cf0978cfe5011845fb43ee63"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:23.269910 21091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:09:23.288515 21091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:09:23.291682 21091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:09:23.291894 21091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:09:23.300660 21091 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:38825
I20260808 02:09:23.300702 21183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:38825 every 8 connection(s)
I20260808 02:09:23.301770 21091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:09:23.304716 21185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:09:23.306469 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 21091
I20260808 02:09:23.306600 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:09:23.306959 5957 external_mini_cluster.cc:1428] Setting key db5896484f23ec723bd07b9f0593127e
I20260808 02:09:23.311911 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: Bootstrap starting.
I20260808 02:09:23.314445 21185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:09:23.315421 21185 log.cc:824] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:09:23 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:09:23.318647 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: No bootstrap required, opened a new log
I20260808 02:09:23.322978 21185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } }
I20260808 02:09:23.323271 21185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:09:23.323334 21185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1b78857815455aa0f69cf175dae1f7, State: Initialized, Role: FOLLOWER
I20260808 02:09:23.323822 21185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } }
I20260808 02:09:23.323979 21185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:09:23.324046 21185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:09:23.324144 21185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:09:23.324931 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:23.308477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.5.209.126:38825)
0808 02:09:23.308954 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:23.308975 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:23.309021 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:09:23.309955 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:23.309982 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:23.310425 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:23.310814 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:09:23.311729 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:23.313220 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:23.314544 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:23.315239 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:23.315412 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:23.318530 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:09:23.318536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:09:23.318553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:09:23.318584 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:09:23.321413 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:23.321927 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:23.321934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:23.321940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:23.322086 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:23.322421 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:23.322425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:23.322426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:23.322930 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:09:23.323160 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:23.323856 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 02:09:23.324164 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260808 02:09:23.325075 21185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } }
I20260808 02:09:23.325480 21185 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1b78857815455aa0f69cf175dae1f7; no voters:
I20260808 02:09:23.325877 21185 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:09:23.325953 21195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:09:23.327164 21185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:09:23.327307 21195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1b78857815455aa0f69cf175dae1f7, State: Running, Role: LEADER
I20260808 02:09:23.327612 21195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } }
I20260808 02:09:23.329645 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1b78857815455aa0f69cf175dae1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } } }
I20260808 02:09:23.329685 21185 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:09:23.329682 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1b78857815455aa0f69cf175dae1f7. Latest consensus state: current_term: 1 leader_uuid: "5c1b78857815455aa0f69cf175dae1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1b78857815455aa0f69cf175dae1f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38825 } } }
I20260808 02:09:23.329790 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:09:23.329841 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:09:23.330265 21213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:09:23.336337 21213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:09:23.343117 21213 catalog_manager.cc:1347] Generated new cluster ID: 3090d3597b134de0b669db1acbca631b
I20260808 02:09:23.343199 21213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:09:23.360337 21213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:09:23.361706 21213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:09:23.368824 21213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: Generated new TSK 0
I20260808 02:09:23.369657 21213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:09:24.082840 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:43931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:09:24.232746 21246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:09:24.233153 21246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:09:24.233297 21246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:09:24.240370 21246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:09:24.240447 21246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:09:24.240541 21246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:09:24.249542 21246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.209.84:43931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:09:24.251173 21246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:09:24.252740 21246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:09:24.265573 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:24.265573 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:24.265573 21266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:24.265925 21246 server_base.cc:1034] running on GCE node
I20260808 02:09:24.266412 21246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:09:24.267112 21246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:09:24.268316 21246 hybrid_clock.cc:648] HybridClock initialized: now 1786154964268281 us; error 59 us; skew 500 ppm
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:09:24.271355 21246 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:09:24.272415 21246 webserver.cc:460] Webserver started at http://127.5.209.65:39527/ using document root <none> and password file <none>
I20260808 02:09:24.273026 21246 fs_manager.cc:359] Metadata directory not provided
I20260808 02:09:24.273126 21246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:09:24.273454 21246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:24.288166 21246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "79943f91c5724e1d9d04dfa109ce5ee6"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "bf7f98066d22af812e57f92aaff160fd"
server_key_iv: "e38e829f2f4de314239e88ae005d8858"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.289321 21246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "79943f91c5724e1d9d04dfa109ce5ee6"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "bf7f98066d22af812e57f92aaff160fd"
server_key_iv: "e38e829f2f4de314239e88ae005d8858"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.295423 21246 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260808 02:09:24.318495 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.320221 21246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:09:24.320355 21246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "79943f91c5724e1d9d04dfa109ce5ee6"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "bf7f98066d22af812e57f92aaff160fd"
server_key_iv: "e38e829f2f4de314239e88ae005d8858"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.320492 21246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:09:24.344343 21246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:09:24.348454 21246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:09:24.348719 21246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:09:24.349517 21246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:09:24.350728 21246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:09:24.350838 21246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.350983 21246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:09:24.351075 21246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.362596 21246 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:36423
I20260808 02:09:24.363927 21246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:09:24.364178 21422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:36423 every 8 connection(s)
I20260808 02:09:24.366513 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 21246
I20260808 02:09:24.366685 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:09:24.367069 5957 external_mini_cluster.cc:1428] Setting key 9555b22c470885ab047dd30085db4ad7
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:09:24.382809 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:24.366446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:41617 (local address 127.5.209.126:38825)
0808 02:09:24.367215 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:24.367225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:24.367741 (+ 516us) server_negotiation.cc:407] Connection header received
0808 02:09:24.369099 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:24.369103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:24.369167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:24.369297 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:09:24.370844 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.371689 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.373717 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.374135 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.374238 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:24.378104 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:09:24.378112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:09:24.378117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:09:24.378153 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:09:24.380213 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:24.380930 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:24.380934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:24.380936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:24.381026 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:24.381540 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:24.381546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:24.381547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:24.381779 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:09:24.381849 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:24.382497 (+ 648us) server_negotiation.cc:299] Negotiation successful
0808 02:09:24.382625 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:09:24.384265 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:24.366902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38825 (local address 127.5.209.65:41617)
0808 02:09:24.367576 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:24.367620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:24.368726 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:24.369534 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:24.369544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:24.369908 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:09:24.370652 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:24.370676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.371844 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.371850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:24.373398 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:24.373413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.374491 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.374498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:24.374745 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:24.375509 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:09:24.375541 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:09:24.377863 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:09:24.380376 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:24.380382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:24.380400 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:24.380802 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:24.381176 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:24.381182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:24.381186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:24.381414 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:24.381911 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:09:24.381916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:09:24.382171 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 02:09:24.382986 (+ 815us) client_negotiation.cc:240] Negotiation successful
0808 02:09:24.383343 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":144,"threads_started":1}
I20260808 02:09:24.387459 21425 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38825
I20260808 02:09:24.387737 21425 heartbeater.cc:464] Registering TS with master...
I20260808 02:09:24.388345 21425 heartbeater.cc:511] Master 127.5.209.126:38825 requested a full tablet report, sending...
I20260808 02:09:24.390333 21136 ts_manager.cc:194] Registered new tserver with Master: 79943f91c5724e1d9d04dfa109ce5ee6 (127.5.209.65:36423)
I20260808 02:09:24.391927 21136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:41617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:09:24.420567 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:43931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:09:24.592481 21434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:09:24.592861 21434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:09:24.592932 21434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:09:24.599011 21434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:09:24.599148 21434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:09:24.599270 21434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:09:24.607208 21434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.209.84:43931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:09:24.608604 21434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:09:24.610033 21434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:09:24.622403 21448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:24.622786 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:24.622861 21434 server_base.cc:1034] running on GCE node
W20260808 02:09:24.623080 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:24.623581 21434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:09:24.624528 21434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:09:24.625733 21434 hybrid_clock.cc:648] HybridClock initialized: now 1786154964625695 us; error 49 us; skew 500 ppm
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:09:24.628707 21434 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:09:24.629696 21434 webserver.cc:460] Webserver started at http://127.5.209.66:35503/ using document root <none> and password file <none>
I20260808 02:09:24.630245 21434 fs_manager.cc:359] Metadata directory not provided
I20260808 02:09:24.630303 21434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:09:24.630474 21434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:24.645690 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "e1f0a8653781441fb23e86f4e31273a9"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "696ee18c93c35dc4cb49f731d1840bb2"
server_key_iv: "177385c73cb2ec8a6fa49422f8e0b35e"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.646616 21434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "e1f0a8653781441fb23e86f4e31273a9"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "696ee18c93c35dc4cb49f731d1840bb2"
server_key_iv: "177385c73cb2ec8a6fa49422f8e0b35e"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.651611 21434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:09:24.672006 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.678484 21434 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260808 02:09:24.678653 21434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "e1f0a8653781441fb23e86f4e31273a9"
format_stamp: "Formatted at 2026-08-08 02:09:24 on dist-test-slave-bw88"
server_key: "696ee18c93c35dc4cb49f731d1840bb2"
server_key_iv: "177385c73cb2ec8a6fa49422f8e0b35e"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:24.678804 21434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:09:24.703485 21434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:09:24.707379 21434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:09:24.707643 21434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:09:24.708436 21434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:09:24.709618 21434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:09:24.709721 21434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.709820 21434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:09:24.709885 21434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:24.723852 21434 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:38557
I20260808 02:09:24.725237 21434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:09:24.726922 21612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:38557 every 8 connection(s)
I20260808 02:09:24.732775 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 21434
I20260808 02:09:24.732913 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:09:24.733268 5957 external_mini_cluster.cc:1428] Setting key 4344cba6b9e977eee163dd1bfbae2198
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:09:24.744772 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:24.729259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:36357 (local address 127.5.209.126:38825)
0808 02:09:24.729446 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:24.729451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:24.730831 (+ 1380us) server_negotiation.cc:407] Connection header received
0808 02:09:24.732088 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:24.732092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:24.732161 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:24.732218 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:09:24.734024 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.734905 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.736555 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.736930 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.737018 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:24.740249 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:09:24.740253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:09:24.740256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:09:24.740282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:09:24.742044 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:24.742779 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:24.742783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:24.742785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:24.742874 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:24.743330 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:24.743334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:24.743336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:24.743575 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:09:24.743669 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:24.744426 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 02:09:24.744580 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:09:24.746778 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:24.730028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38825 (local address 127.5.209.66:36357)
0808 02:09:24.730646 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:24.730704 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:24.731825 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:24.732630 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:24.732639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:24.732994 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:09:24.733837 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:24.733860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.735073 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.735076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:24.736413 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:24.736425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:24.737246 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:24.737250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:24.737480 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:24.738180 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:09:24.738204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:09:24.740034 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:09:24.742269 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:24.742276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:24.742293 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:24.742655 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:24.743001 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:24.743004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:24.743006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:24.743207 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:24.743745 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:09:24.743752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:09:24.744089 (+ 337us) client_negotiation.cc:769] Sending connection context
0808 02:09:24.745274 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0808 02:09:24.745672 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":163,"threads_started":1}
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:09:24.748708 21613 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38825
I20260808 02:09:24.748970 21613 heartbeater.cc:464] Registering TS with master...
I20260808 02:09:24.749665 21613 heartbeater.cc:511] Master 127.5.209.126:38825 requested a full tablet report, sending...
I20260808 02:09:24.751032 21136 ts_manager.cc:194] Registered new tserver with Master: e1f0a8653781441fb23e86f4e31273a9 (127.5.209.66:38557)
I20260808 02:09:24.751834 21136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:36357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:09:24.777221 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:43931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
W20260808 02:09:24.949599 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:09:24.950031 21620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:09:24.950187 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:09:24.956154 21620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:09:24.956250 21620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:09:24.956357 21620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:09:24.970691 21620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.209.84:43931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:09:24.972566 21620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:09:24.974315 21620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:09:24.987870 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:24.988160 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:09:24.988304 21629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:09:24.988831 21620 server_base.cc:1034] running on GCE node
I20260808 02:09:24.989411 21620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:09:24.990302 21620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:09:24.991523 21620 hybrid_clock.cc:648] HybridClock initialized: now 1786154964991496 us; error 38 us; skew 500 ppm
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:09:24.995505 21620 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:09:24.996695 21620 webserver.cc:460] Webserver started at http://127.5.209.67:37873/ using document root <none> and password file <none>
I20260808 02:09:24.997473 21620 fs_manager.cc:359] Metadata directory not provided
I20260808 02:09:24.997561 21620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:09:24.997864 21620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:09:24 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:25.018268 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "0f6459ad656e401f96104a297ac1d56d"
format_stamp: "Formatted at 2026-08-08 02:09:25 on dist-test-slave-bw88"
server_key: "8f6d177b638bfc01e1c31967a0ea38f2"
server_key_iv: "9bb492685bac9a379029e173415b9ac7"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:25.019014 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "0f6459ad656e401f96104a297ac1d56d"
format_stamp: "Formatted at 2026-08-08 02:09:25 on dist-test-slave-bw88"
server_key: "8f6d177b638bfc01e1c31967a0ea38f2"
server_key_iv: "9bb492685bac9a379029e173415b9ac7"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:25.023471 21620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:09:25.038657 21643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:25.040165 21620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:09:25.040323 21620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "0f6459ad656e401f96104a297ac1d56d"
format_stamp: "Formatted at 2026-08-08 02:09:25 on dist-test-slave-bw88"
server_key: "8f6d177b638bfc01e1c31967a0ea38f2"
server_key_iv: "9bb492685bac9a379029e173415b9ac7"
server_key_version: "kuduclusterkey@0"
I20260808 02:09:25.040474 21620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:09:25.069708 21620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:09:25.073601 21620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:09:25.073904 21620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:09:25.074888 21620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:09:25.076022 21620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:09:25.076133 21620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:25.076236 21620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:09:25.076303 21620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:09:25.093089 21620 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:43087
I20260808 02:09:25.094442 21620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:09:25.099901 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 21620
I20260808 02:09:25.100175 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:09:25.100639 5957 external_mini_cluster.cc:1428] Setting key a5473d5149a1d62bcbe9334d8ac012d8
I20260808 02:09:25.096607 21784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:43087 every 8 connection(s)
Aug 08 02:09:25 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:09:25.119823 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.098874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59585 (local address 127.5.209.126:38825)
0808 02:09:25.099015 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.099020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.103342 (+ 4322us) server_negotiation.cc:407] Connection header received
0808 02:09:25.104678 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.104682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.104736 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.104783 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:09:25.106532 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.107299 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.108970 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.109285 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.109368 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.112602 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:09:25.112609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:09:25.112613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:09:25.112648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:09:25.116990 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:25.117864 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:25.117868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:25.117870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:25.117969 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:25.118444 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:25.118447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:25.118449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:25.118707 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:09:25.118799 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.119522 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.119655 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:09:25.121809 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.099652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:38825 (local address 127.5.209.67:59585)
0808 02:09:25.103192 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.103251 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.104422 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.105233 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.105242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.105624 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:09:25.106361 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.106381 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.107453 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.107456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.108814 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.108826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.109577 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.109580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.109775 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.110448 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:09:25.110478 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:09:25.112363 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:09:25.117256 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:25.117265 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:25.117310 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:25.117729 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:25.118108 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:09:25.118114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:09:25.118117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:09:25.118323 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:09:25.118881 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:09:25.118888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:09:25.119220 (+ 332us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.120283 (+ 1063us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.120693 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3296,"thread_start_us":143,"threads_started":1}
I20260808 02:09:25.123270 21785 heartbeater.cc:347] Connected to a master server at 127.5.209.126:38825
I20260808 02:09:25.123525 21785 heartbeater.cc:464] Registering TS with master...
I20260808 02:09:25.124257 21785 heartbeater.cc:511] Master 127.5.209.126:38825 requested a full tablet report, sending...
I20260808 02:09:25.126143 21134 ts_manager.cc:194] Registered new tserver with Master: 0f6459ad656e401f96104a297ac1d56d (127.5.209.67:43087)
I20260808 02:09:25.127018 21134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:59585
I20260808 02:09:25.128578 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:09:25.145433 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.133638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.5.209.126:38825)
0808 02:09:25.133766 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.133771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.134497 (+ 726us) server_negotiation.cc:407] Connection header received
0808 02:09:25.134706 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.134710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.134759 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.134812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:09:25.135827 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.136681 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.138057 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.138420 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.138526 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.140655 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:09:25.140660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:09:25.140663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:09:25.140695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:09:25.142880 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:25.143583 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:25.143587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:25.143589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:25.143690 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:09:25.144105 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:09:25.144110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:09:25.144112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:09:25.144438 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:09:25.144556 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.145078 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.145216 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:09:25.149904 21136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:09:25.154873 21136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:09:25.187891 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.168611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:36423 (local address 127.0.0.1:60772)
0808 02:09:25.174856 (+ 6245us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.174982 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.175140 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.182269 (+ 7129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.182273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.182292 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:09:25.183028 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.183041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.184849 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.184853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.185993 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.186005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.187278 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.187282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.187412 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.187473 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.187603 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.187727 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5699,"thread_start_us":189,"threads_started":1}
I20260808 02:09:25.188707 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.168790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.5.209.65:36423)
0808 02:09:25.181584 (+ 12794us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.181592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.181620 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:09:25.181709 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.181715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.181928 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.182102 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:09:25.183182 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.184605 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.186145 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.186945 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.187061 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.187206 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.188417 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.188541 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12660,"thread_start_us":125,"threads_started":1}
I20260808 02:09:25.188884 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.171347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:43087 (local address 127.0.0.1:42546)
0808 02:09:25.175374 (+ 4027us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.175386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.175461 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.176973 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.176984 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.177021 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:09:25.177162 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.177168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.179785 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.179789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.184598 (+ 4809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.184610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.188255 (+ 3645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.188259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.188580 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.188604 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.188668 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.188751 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3964,"thread_start_us":83,"threads_started":1}
I20260808 02:09:25.189669 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.172612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.5.209.66:38557)
0808 02:09:25.174206 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.174228 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.179202 (+ 4974us) server_negotiation.cc:407] Connection header received
0808 02:09:25.179285 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.179290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.179499 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.179641 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:09:25.181593 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.182716 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.187165 (+ 4449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.187945 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.188061 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.188192 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.189428 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.189533 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":69,"threads_started":1}
I20260808 02:09:25.191416 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.172244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.5.209.67:43087)
0808 02:09:25.174714 (+ 2470us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.174733 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.175554 (+ 821us) server_negotiation.cc:407] Connection header received
0808 02:09:25.175642 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.175648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.175881 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.176063 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:09:25.177609 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.178747 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.185634 (+ 6887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.186446 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.186560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.186719 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.191160 (+ 4441us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.191274 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2298,"thread_start_us":101,"threads_started":1}
I20260808 02:09:25.191653 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.171661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:38557 (local address 127.0.0.1:35858)
0808 02:09:25.178968 (+ 7307us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.178981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.179090 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.180823 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.180826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.180844 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:09:25.181451 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.181458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.182862 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.182866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.187008 (+ 4142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.187019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.189149 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.189153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.189235 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.189257 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.189313 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.190238 (+ 925us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7244,"mutex_wait_us":502,"thread_start_us":86,"threads_started":1}
I20260808 02:09:25.192520 21328 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4865ac8066461e81138d0b5a434813 (DEFAULT_TABLE table=test-table [id=78a94aaea0d94e9b8565667eff70ecb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:09:25.193900 21328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4865ac8066461e81138d0b5a434813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:09:25.194929 21522 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4865ac8066461e81138d0b5a434813 (DEFAULT_TABLE table=test-table [id=78a94aaea0d94e9b8565667eff70ecb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:09:25.196110 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4865ac8066461e81138d0b5a434813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:09:25.208221 21707 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4865ac8066461e81138d0b5a434813 (DEFAULT_TABLE table=test-table [id=78a94aaea0d94e9b8565667eff70ecb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:09:25.209825 21814 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Bootstrap starting.
I20260808 02:09:25.210359 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4865ac8066461e81138d0b5a434813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:09:25.212694 21814 tablet_bootstrap.cc:654] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Neither blocks nor log segments found. Creating new log.
I20260808 02:09:25.213840 21814 log.cc:824] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:09:25.221377 21815 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Bootstrap starting.
I20260808 02:09:25.224647 21815 tablet_bootstrap.cc:654] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Neither blocks nor log segments found. Creating new log.
I20260808 02:09:25.225232 21818 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Bootstrap starting.
I20260808 02:09:25.227993 21814 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: No bootstrap required, opened a new log
I20260808 02:09:25.228431 21815 log.cc:824] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:09:25.228510 21814 ts_tablet_manager.cc:1397] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260808 02:09:25.230057 21818 tablet_bootstrap.cc:654] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Neither blocks nor log segments found. Creating new log.
I20260808 02:09:25.231137 21818 log.cc:824] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:09:25.233982 21815 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: No bootstrap required, opened a new log
I20260808 02:09:25.234150 21814 raft_consensus.cc:348] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.234637 21814 raft_consensus.cc:374] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:09:25.234759 21814 raft_consensus.cc:729] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79943f91c5724e1d9d04dfa109ce5ee6, State: Initialized, Role: FOLLOWER
I20260808 02:09:25.235415 21815 ts_tablet_manager.cc:1397] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260808 02:09:25.235401 21814 consensus_queue.cc:260] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.236778 21814 ts_tablet_manager.cc:1428] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 02:09:25.237242 21425 heartbeater.cc:503] Master 127.5.209.126:38825 was elected leader, sending a full tablet report...
I20260808 02:09:25.240298 21815 raft_consensus.cc:348] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.240725 21815 raft_consensus.cc:374] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:09:25.240824 21815 raft_consensus.cc:729] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f0a8653781441fb23e86f4e31273a9, State: Initialized, Role: FOLLOWER
I20260808 02:09:25.241456 21815 consensus_queue.cc:260] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.243404 21815 ts_tablet_manager.cc:1428] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:09:25.245095 21818 tablet_bootstrap.cc:492] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: No bootstrap required, opened a new log
I20260808 02:09:25.245481 21818 ts_tablet_manager.cc:1397] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260808 02:09:25.251230 21818 raft_consensus.cc:348] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.251597 21818 raft_consensus.cc:374] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:09:25.251693 21818 raft_consensus.cc:729] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6459ad656e401f96104a297ac1d56d, State: Initialized, Role: FOLLOWER
I20260808 02:09:25.252271 21818 consensus_queue.cc:260] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.253540 21613 heartbeater.cc:503] Master 127.5.209.126:38825 was elected leader, sending a full tablet report...
I20260808 02:09:25.256311 21818 ts_tablet_manager.cc:1428] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260808 02:09:25.256724 21785 heartbeater.cc:503] Master 127.5.209.126:38825 was elected leader, sending a full tablet report...
Aug 08 02:09:25 dist-test-slave-bw88 krb5kdc[20460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154964, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:09:25.302716 21825 raft_consensus.cc:482] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:09:25.302955 21825 raft_consensus.cc:504] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.304394 21825 leader_election.cc:288] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79943f91c5724e1d9d04dfa109ce5ee6 (127.5.209.65:36423), e1f0a8653781441fb23e86f4e31273a9 (127.5.209.66:38557)
I20260808 02:09:25.310456 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.304631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:36423 (local address 127.5.209.67:59425)
0808 02:09:25.305448 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.305465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.305569 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.306388 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.306391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.306414 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:09:25.306551 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.306559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.307649 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.307653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.308979 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.308992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.309981 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.309985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.310088 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.310112 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.310233 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.310315 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723}
I20260808 02:09:25.311547 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.305865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:59425 (local address 127.5.209.65:36423)
0808 02:09:25.306027 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.306032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.306051 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:09:25.306112 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.306115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.306167 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.306257 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:09:25.306696 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.307484 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.309131 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.309752 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.309867 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.309923 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.311328 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.311412 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:09:25.312182 21360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4865ac8066461e81138d0b5a434813" candidate_uuid: "0f6459ad656e401f96104a297ac1d56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" is_pre_election: true
I20260808 02:09:25.312547 21360 raft_consensus.cc:2454] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6459ad656e401f96104a297ac1d56d in term 0.
I20260808 02:09:25.313119 21644 leader_election.cc:302] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6459ad656e401f96104a297ac1d56d, 79943f91c5724e1d9d04dfa109ce5ee6; no voters:
I20260808 02:09:25.313483 21825 raft_consensus.cc:2781] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:09:25.313597 21825 raft_consensus.cc:482] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:09:25.313659 21825 raft_consensus.cc:3037] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:09:25.315137 21825 raft_consensus.cc:504] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.315933 21360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4865ac8066461e81138d0b5a434813" candidate_uuid: "0f6459ad656e401f96104a297ac1d56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79943f91c5724e1d9d04dfa109ce5ee6"
I20260808 02:09:25.316066 21360 raft_consensus.cc:3037] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:09:25.317456 21360 raft_consensus.cc:2454] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6459ad656e401f96104a297ac1d56d in term 1.
I20260808 02:09:25.317818 21644 leader_election.cc:302] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6459ad656e401f96104a297ac1d56d, 79943f91c5724e1d9d04dfa109ce5ee6; no voters:
I20260808 02:09:25.320771 21825 leader_election.cc:288] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [CANDIDATE]: Term 1 election: Requested vote from peers 79943f91c5724e1d9d04dfa109ce5ee6 (127.5.209.65:36423), e1f0a8653781441fb23e86f4e31273a9 (127.5.209.66:38557)
I20260808 02:09:25.320988 21825 raft_consensus.cc:2781] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:09:25.321110 21825 raft_consensus.cc:686] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 1 LEADER]: Becoming Leader. State: Replica: 0f6459ad656e401f96104a297ac1d56d, State: Running, Role: LEADER
I20260808 02:09:25.321594 21825 consensus_queue.cc:237] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } }
I20260808 02:09:25.323496 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.306033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:37533 (local address 127.5.209.66:38557)
0808 02:09:25.306172 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.306177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.310998 (+ 4821us) server_negotiation.cc:407] Connection header received
0808 02:09:25.311065 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.311069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.311128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.311206 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:09:25.318443 (+ 7237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.319269 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.321857 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.322480 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.322596 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.322653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.323290 (+ 637us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.323381 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:09:25.323190 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.304922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:38557 (local address 127.5.209.67:37533)
0808 02:09:25.310697 (+ 5775us) negotiation.cc:107] Waiting for socket to connect
0808 02:09:25.310712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:09:25.310792 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:09:25.318135 (+ 7343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:09:25.318139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:09:25.318175 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:09:25.318270 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.318277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.319413 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.319416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.320562 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:09:25.320573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.322766 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.322770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:09:25.322864 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.322887 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:09:25.322944 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:09:25.323024 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5717,"thread_start_us":189,"threads_started":1}
I20260808 02:09:25.325682 21529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4865ac8066461e81138d0b5a434813" candidate_uuid: "0f6459ad656e401f96104a297ac1d56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f0a8653781441fb23e86f4e31273a9" is_pre_election: true
I20260808 02:09:25.326022 21529 raft_consensus.cc:2454] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6459ad656e401f96104a297ac1d56d in term 0.
I20260808 02:09:25.326381 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4865ac8066461e81138d0b5a434813" candidate_uuid: "0f6459ad656e401f96104a297ac1d56d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f0a8653781441fb23e86f4e31273a9"
I20260808 02:09:25.326524 21549 raft_consensus.cc:3037] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:09:25.328054 21549 raft_consensus.cc:2454] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6459ad656e401f96104a297ac1d56d in term 1.
I20260808 02:09:25.331907 21136 catalog_manager.cc:5495] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6459ad656e401f96104a297ac1d56d (127.5.209.67). New cstate: current_term: 1 leader_uuid: "0f6459ad656e401f96104a297ac1d56d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6459ad656e401f96104a297ac1d56d" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 } health_report { overall_health: UNKNOWN } } }
W20260808 02:09:25.348764 21788 tablet.cc:2369] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:09:25.365935 21426 tablet.cc:2369] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:09:25.396688 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:09:25.382591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.5.209.67:43087)
0808 02:09:25.382720 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:09:25.382724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:09:25.382738 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:09:25.382841 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:09:25.382844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:09:25.382912 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:09:25.383024 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:09:25.383590 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.384410 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.394144 (+ 9734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:09:25.394528 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:09:25.394657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:09:25.395177 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:09:25.395804 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:09:25.395928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:09:25.396429 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 02:09:25.396512 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:09:25.413836 21549 raft_consensus.cc:1261] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 1 FOLLOWER]: Refusing update from remote peer 0f6459ad656e401f96104a297ac1d56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:09:25.414887 21360 raft_consensus.cc:1261] T 9d4865ac8066461e81138d0b5a434813 P 79943f91c5724e1d9d04dfa109ce5ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0f6459ad656e401f96104a297ac1d56d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:09:25.415823 21828 consensus_queue.cc:1035] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f0a8653781441fb23e86f4e31273a9" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:09:25.416335 21828 consensus_queue.cc:1035] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79943f91c5724e1d9d04dfa109ce5ee6" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:09:25.436051 21835 mvcc.cc:205] Tried to move back new op lower bound from 7316090738309300224 to 7316090737987817472. Current Snapshot: MvccSnapshot[applied={T|T < 7316090738309300224}]
I20260808 02:09:25.446338 21837 mvcc.cc:205] Tried to move back new op lower bound from 7316090738309300224 to 7316090737987817472. Current Snapshot: MvccSnapshot[applied={T|T < 7316090738309300224}]
I20260808 02:09:25.454279 21136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:09:25.454512 21136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:09:25.457474 21136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c1b78857815455aa0f69cf175dae1f7: Sending DeleteTablet for 3 replicas of tablet 9d4865ac8066461e81138d0b5a434813
I20260808 02:09:25.458524 21328 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4865ac8066461e81138d0b5a434813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:09:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:60772
I20260808 02:09:25.459354 21522 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4865ac8066461e81138d0b5a434813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:09:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:35858
I20260808 02:09:25.459977 21707 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4865ac8066461e81138d0b5a434813 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:09:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:42546
I20260808 02:09:25.461643 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 21246
I20260808 02:09:25.509887 21848 tablet_replica.cc:330] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: stopping tablet replica
I20260808 02:09:25.510427 21848 raft_consensus.cc:2229] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:09:25.510836 21850 tablet_replica.cc:330] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: stopping tablet replica
I20260808 02:09:25.511350 21848 raft_consensus.cc:2258] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:09:25.511734 21850 raft_consensus.cc:2229] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:09:25.517871 21850 raft_consensus.cc:2258] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:09:25.519948 21848 ts_tablet_manager.cc:1905] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:09:25.521636 21850 ts_tablet_manager.cc:1905] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:09:25.524704 21848 ts_tablet_manager.cc:1918] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:09:25.531584 21850 ts_tablet_manager.cc:1918] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:09:25.531790 21848 log.cc:1196] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/9d4865ac8066461e81138d0b5a434813
I20260808 02:09:25.531941 21850 log.cc:1196] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/9d4865ac8066461e81138d0b5a434813
I20260808 02:09:25.532405 21848 ts_tablet_manager.cc:1939] T 9d4865ac8066461e81138d0b5a434813 P e1f0a8653781441fb23e86f4e31273a9: Deleting consensus metadata
I20260808 02:09:25.532806 21850 ts_tablet_manager.cc:1939] T 9d4865ac8066461e81138d0b5a434813 P 0f6459ad656e401f96104a297ac1d56d: Deleting consensus metadata
I20260808 02:09:25.534471 21108 catalog_manager.cc:4841] TS e1f0a8653781441fb23e86f4e31273a9 (127.5.209.66:38557): tablet 9d4865ac8066461e81138d0b5a434813 (table test-table [id=78a94aaea0d94e9b8565667eff70ecb0]) successfully deleted
I20260808 02:09:25.535904 21108 catalog_manager.cc:4841] TS 0f6459ad656e401f96104a297ac1d56d (127.5.209.67:43087): tablet 9d4865ac8066461e81138d0b5a434813 (table test-table [id=78a94aaea0d94e9b8565667eff70ecb0]) successfully deleted
W20260808 02:09:25.546705 21108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:36423 (error 108)
I20260808 02:09:25.554795 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 21434
I20260808 02:09:25.589614 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 21620
I20260808 02:09:25.610746 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 21091
I20260808 02:09:25.634476 5957 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:09:25.990056 5957 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:09:25.990186 5957 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:09:26.335309 5957 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:09:26.335 UTC [20499] LOG: received smart shutdown request
2026-08-08 02:09:26.339 UTC [20499] LOG: background worker "logical replication launcher" (PID 20510) exited with exit code 1
2026-08-08 02:09:26.340 UTC [20503] LOG: shutting down
2026-08-08 02:09:26.508 UTC [20499] LOG: database system is shut down
2026-08-08T02:09:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49457 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:09:26 dist-test-slave-bw88 krb5kdc[21861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:09:26 dist-test-slave-bw88 krb5kdc[21861](info): set up 2 sockets
Aug 08 02:09:26 dist-test-slave-bw88 krb5kdc[21861](info): commencing operation
krb5kdc: starting...
W20260808 02:09:27.833734 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:09:27 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786154967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:09:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangeradmin_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangeradmin_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangerlookup_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangerlookup_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
I20260808 02:09:28.014055 5957 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskguwdHL/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/postgres -l logfile start
2026-08-08 02:09:30.511 UTC [21899] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:09:30.512 UTC [21899] LOG: listening on IPv4 address "127.5.209.84", port 57159
2026-08-08 02:09:30.515 UTC [21899] LOG: listening on Unix socket "/tmp/.s.PGSQL.57159"
2026-08-08 02:09:30.537 UTC [21905] LOG: database system was shut down at 2026-08-08 02:09:29 UTC
2026-08-08 02:09:30.541 UTC [21899] LOG: database system is ready to accept connections
I20260808 02:09:31.674484 5957 mini_postgres.cc:96] Postgres bound to 57159
2026-08-08 02:09:31.682 UTC [21917] FATAL: database "slave" does not exist
127.5.209.84:57159 - accepting connections
I20260808 02:09:31.683447 5957 mini_ranger.cc:161] Starting Ranger...
I20260808 02:09:31.700337 5957 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:09:32.166050 5957 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:09:32.166147 5957 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin
2026-08-08 02:09:32,455 [I] DB FLAVOR :POSTGRES
2026-08-08 02:09:32,456 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:09:32,456 [I] Checking connection..
2026-08-08 02:09:32,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:09:32,727 [I] Checking connection passed.
2026-08-08 02:09:32,728 [I] --------- Verifying version history table ---------
2026-08-08 02:09:32,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:09:32,982 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:09:32,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:09:33,241 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:09:33,242 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:09:33,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:09:33.491 UTC [22000] WARNING: there is no transaction in progress
2026-08-08 02:09:33,499 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:09:33,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:09:33,766 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:09:33,767 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:09:33,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:09:34,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:09:34,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bw88.c.gcp-upstream.internal','N') ;"
2026-08-08 02:09:34,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:09:34,825 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:09:34,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:09:35,104 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:09:35,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:09:35,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:09:35,589 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:09:35,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:09:36.610 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.626 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.642 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.659 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.675 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.693 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.749 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.755 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.770 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.780 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.790 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.804 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.814 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.819 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.825 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:36.832 UTC [22144] WARNING: there is no transaction in progress
2026-08-08 02:09:37,472 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:09:37,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:09:37,722 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:09:37,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:09:37,986 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:09:37,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:09:38,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:09:38,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:09:38,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:09:38,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bw88.c.gcp-upstream.internal';"
2026-08-08 02:09:39,034 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:09:39,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:09:39,286 [I] DB_PATCHES have already been applied
I20260808 02:09:39.292435 5957 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskguwdHL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:09:39.292523 5957 mini_ranger.cc:193] Using host: 127.5.209.84
I20260808 02:09:39.297034 5957 mini_ranger.cc:227] Ranger admin URL: http://127.5.209.84:35493
Aug 08, 2026 2:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:09:40.583928 5957 mini_ranger.cc:160] Time spent starting Ranger: real 8.900s user 0.003s sys 0.010s
Aug 08, 2026 2:10:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:10:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 2:10:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:10:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:10:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:10:02.984202 5957 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.209.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.209.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangerkms_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.209.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/rangerkms_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.209.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.209.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.209.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/HTTP_127.5.209.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:10:03 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155003, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:10:03.081624 5957 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:10:03.096428 5957 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:10:04.003750 5957 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:10:04.003849 5957 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms
I20260808 02:10:04.003870 5957 mini_ranger_kms.cc:227] Using postgres at 127.5.209.84:57159
2026-08-08 02:10:04,194 [I] DB FLAVOR :POSTGRES
2026-08-08 02:10:04,194 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:10:04,194 [I] Checking connection
2026-08-08 02:10:04,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:10:04,467 [I] connection success
2026-08-08 02:10:04,468 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:10:04,468 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:10:04,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:10:04,754 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:10:04,754 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:10:04,754 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:10:04,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.209.84:57159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:10:05,037 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:10:05.500720 5957 mini_ranger_kms.cc:310] Created kms service
I20260808 02:10:05.685302 5957 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:10:05.729871 5957 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:10:05 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155003, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:06.138710 5957 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:10:06.138870 5957 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms:/tmp/dist-test-taskguwdHL/build/debug/bin/postgresql.jar:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskguwdHL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:10:06.138891 5957 mini_ranger_kms.cc:240] Using host: 127.5.209.84
I20260808 02:10:06.147881 5957 mini_ranger_kms.cc:276] Ranger KMS PID: 22387
I20260808 02:10:06.147969 5957 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.209.84:59811
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:10:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:10:08 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:08.868203 5957 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.787s user 0.003s sys 0.175s
I20260808 02:10:08.868346 5957 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:10:08.868423 5957 mini_ranger_kms.cc:401] 127.5.209.84:59811/kms/v1/keys
Aug 08 02:10:09 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
Aug 08 02:10:09 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.209.84@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
Aug 08 02:10:11 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:10:11.873296 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33321
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33321
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:11.993623 22463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:11.993880 22463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:11.993930 22463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:11.997938 22463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:11.998004 22463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:11.998024 22463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:11.998041 22463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:11.998057 22463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:12.003682 22463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.209.84:59811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33321
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:12.004801 22463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:12.005944 22463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:12.015115 22476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:12.015143 22473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:12.015146 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:12.015347 22463 server_base.cc:1034] running on GCE node
I20260808 02:10:12.016209 22463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:12.017623 22463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:12.018827 22463 hybrid_clock.cc:648] HybridClock initialized: now 1786155012018796 us; error 40 us; skew 500 ppm
Aug 08 02:10:12 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155012, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:12.027004 22463 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:12.028053 22463 webserver.cc:460] Webserver started at http://127.5.209.126:39751/ using document root <none> and password file <none>
I20260808 02:10:12.028637 22463 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:12.028717 22463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:12.028955 22463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:10:12 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155012, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:12.082432 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "05b5bccc57f64ce8bc5658fd064d2ee9"
format_stamp: "Formatted at 2026-08-08 02:10:12 on dist-test-slave-bw88"
server_key: "2a27043e36a0c1c390039bc0ba394d5c"
server_key_iv: "d40280ab1ff7a1932ba7e18360fbdc60"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:12.083103 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "05b5bccc57f64ce8bc5658fd064d2ee9"
format_stamp: "Formatted at 2026-08-08 02:10:12 on dist-test-slave-bw88"
server_key: "2a27043e36a0c1c390039bc0ba394d5c"
server_key_iv: "d40280ab1ff7a1932ba7e18360fbdc60"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:12.087435 22463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:10:12.105378 22485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:12.107142 22463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:10:12.107306 22463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "05b5bccc57f64ce8bc5658fd064d2ee9"
format_stamp: "Formatted at 2026-08-08 02:10:12 on dist-test-slave-bw88"
server_key: "2a27043e36a0c1c390039bc0ba394d5c"
server_key_iv: "d40280ab1ff7a1932ba7e18360fbdc60"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:12.107466 22463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:12.141218 22463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:12.144678 22463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:12.144881 22463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:12.153785 22463 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33321
I20260808 02:10:12.153800 22551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33321 every 8 connection(s)
I20260808 02:10:12.155004 22463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:12.158113 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:12.163060 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 22463
I20260808 02:10:12.163188 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:10:12.163493 5957 external_mini_cluster.cc:1428] Setting key 000d2e141c8aebe9ba29b1ea90136776
I20260808 02:10:12.163935 22552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Bootstrap starting.
I20260808 02:10:12.166808 22552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:12.167753 22552 log.cc:824] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:12.170504 22552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: No bootstrap required, opened a new log
Aug 08 02:10:12 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:12.175467 22552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:12.175798 22552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:12.175904 22552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5bccc57f64ce8bc5658fd064d2ee9, State: Initialized, Role: FOLLOWER
I20260808 02:10:12.176425 22552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:12.176635 22552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:12.176707 22552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:12.176838 22552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:12.178033 22552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:12.178414 22552 leader_election.cc:302] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b5bccc57f64ce8bc5658fd064d2ee9; no voters:
I20260808 02:10:12.178788 22552 leader_election.cc:288] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:10:12.178897 22557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:12.178997 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:12.165043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.5.209.126:33321)
0808 02:10:12.165577 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:12.165597 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:12.165652 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:10:12.166460 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:12.166484 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:12.166926 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:12.167307 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:12.168289 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:12.169548 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:12.170440 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:12.170992 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:12.171174 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:12.173842 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:12.173848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:12.173862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:12.173889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:12.175607 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:12.176115 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:12.176121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:12.176126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:12.176283 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:12.176571 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:12.176653 (+ 82us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:12.176655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:12.177180 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:12.177424 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:12.177703 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 02:10:12.178017 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":150,"threads_started":1}
I20260808 02:10:12.179186 22557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 05b5bccc57f64ce8bc5658fd064d2ee9, State: Running, Role: LEADER
I20260808 02:10:12.180330 22552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:12.180680 22557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:12.182704 22552 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:10:12.182792 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } } }
I20260808 02:10:12.182953 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:12.182881 22559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b5bccc57f64ce8bc5658fd064d2ee9. Latest consensus state: current_term: 1 leader_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } } }
I20260808 02:10:12.183207 22559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:12.183509 22574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:12.191535 22574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:12.199079 22574 catalog_manager.cc:1347] Generated new cluster ID: 7da78fc08d29432ea9626306a4f8ea3f
I20260808 02:10:12.199141 22574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:12.234377 22574 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:10:12.235910 22574 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:12.242223 22574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Generated new TSK 0
I20260808 02:10:12.243265 22574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:10:13.002415 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:13.134490 22612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:13.134917 22612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:13.135026 22612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:13.141109 22612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:13.141211 22612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:13.141309 22612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:13.149444 22612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:13.150988 22612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:13.152478 22612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:13.165421 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:13.165506 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:13.165802 22630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:13.166611 22612 server_base.cc:1034] running on GCE node
I20260808 02:10:13.167127 22612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:13.168020 22612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:13.169235 22612 hybrid_clock.cc:648] HybridClock initialized: now 1786155013169214 us; error 35 us; skew 500 ppm
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:13.173079 22612 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:13.174238 22612 webserver.cc:460] Webserver started at http://127.5.209.65:46153/ using document root <none> and password file <none>
I20260808 02:10:13.175012 22612 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:13.175110 22612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:13.175436 22612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:13.192474 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "75ca753344c1484fac1af38b8ded5935"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "5d0c57b15e5dbcee8c1d8d037311dd39"
server_key_iv: "87a79c6c4e93cdae3a72ad7607a7f792"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.193416 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "75ca753344c1484fac1af38b8ded5935"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "5d0c57b15e5dbcee8c1d8d037311dd39"
server_key_iv: "87a79c6c4e93cdae3a72ad7607a7f792"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.199365 22612 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260808 02:10:13.237578 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.239526 22612 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260808 02:10:13.239672 22612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "75ca753344c1484fac1af38b8ded5935"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "5d0c57b15e5dbcee8c1d8d037311dd39"
server_key_iv: "87a79c6c4e93cdae3a72ad7607a7f792"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.239818 22612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:13.272825 22612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:13.276791 22612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:13.277086 22612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:13.277976 22612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:13.279203 22612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:13.279302 22612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.279393 22612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:13.279454 22612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.291823 22612 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:41531
I20260808 02:10:13.293198 22612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:13.293900 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 22612
I20260808 02:10:13.294042 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:10:13.294385 5957 external_mini_cluster.cc:1428] Setting key 77267d9b747796c4a637a729593bf713
I20260808 02:10:13.304112 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:41531 every 8 connection(s)
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:13.340463 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:13.307032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:43803 (local address 127.5.209.126:33321)
0808 02:10:13.312397 (+ 5365us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:13.312402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:13.312419 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:10:13.312467 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:13.312470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:13.312532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:13.312628 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:13.315745 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.316605 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.318439 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.318811 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.318922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:13.330854 (+ 11932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:13.330859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:13.330862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:13.330900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:13.333534 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:13.336548 (+ 3014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:13.336553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:13.336554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:13.336654 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:13.337147 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:13.337151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:13.337153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:13.337465 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:13.337587 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:13.339368 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0808 02:10:13.340263 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5276,"thread_start_us":106,"threads_started":1}
I20260808 02:10:13.343206 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:13.307600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.65:43803)
0808 02:10:13.308957 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:13.309022 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:13.310192 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:13.312957 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:13.312967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:13.313454 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:13.314191 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:13.314216 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.316772 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.316777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:13.318285 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:13.318299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.324431 (+ 6132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.324436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:13.324693 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:13.325474 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:13.325505 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:13.330566 (+ 5061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:13.333698 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:13.335903 (+ 2205us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:13.335927 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:13.336386 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:13.336780 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:13.336783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:13.336785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:13.337019 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:13.337652 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:13.337659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:13.338059 (+ 400us) client_negotiation.cc:769] Sending connection context
0808 02:10:13.338293 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 02:10:13.338703 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":154,"threads_started":1}
I20260808 02:10:13.354132 22788 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:13.354444 22788 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:13.355159 22788 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:13.357183 22504 ts_manager.cc:194] Registered new tserver with Master: 75ca753344c1484fac1af38b8ded5935 (127.5.209.65:41531)
I20260808 02:10:13.358871 22504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:43803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:10:13.387092 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:13.558817 22795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:13.559177 22795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:13.559250 22795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:13.565310 22795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:13.565464 22795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:13.565572 22795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:13.573016 22795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:13.574734 22795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:13.576167 22795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:13.591948 22806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:13.593832 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:13.594266 22808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:13.595278 22795 server_base.cc:1034] running on GCE node
I20260808 02:10:13.595975 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:13.596925 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:13.601013 22795 hybrid_clock.cc:648] HybridClock initialized: now 1786155013600932 us; error 88 us; skew 500 ppm
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:13.606484 22795 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:13.607657 22795 webserver.cc:460] Webserver started at http://127.5.209.66:37911/ using document root <none> and password file <none>
I20260808 02:10:13.608294 22795 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:13.608397 22795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:13.608670 22795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:13.626247 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "9398a041c217426e86a2e1546bf8cbf2"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "427858add18119cd473d12bdfb6bd9cd"
server_key_iv: "06b69ba68d283ec3615162c36f5b48ce"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.627259 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "9398a041c217426e86a2e1546bf8cbf2"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "427858add18119cd473d12bdfb6bd9cd"
server_key_iv: "06b69ba68d283ec3615162c36f5b48ce"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.632138 22795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:10:13.677806 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.681775 22795 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.000s
I20260808 02:10:13.682047 22795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "9398a041c217426e86a2e1546bf8cbf2"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "427858add18119cd473d12bdfb6bd9cd"
server_key_iv: "06b69ba68d283ec3615162c36f5b48ce"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:13.682430 22795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:13.711292 22795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:13.715869 22795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:13.716192 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:13.717121 22795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:13.718395 22795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:13.718521 22795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.718643 22795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:13.718765 22795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:13.735944 22795 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:34063
I20260808 02:10:13.737416 22795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:13.738474 22964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:34063 every 8 connection(s)
I20260808 02:10:13.743466 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 22795
I20260808 02:10:13.743646 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:10:13.744055 5957 external_mini_cluster.cc:1428] Setting key 68527287fbab33e76d173897d141f3e7
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:13.766644 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:13.743047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:33421 (local address 127.5.209.126:33321)
0808 02:10:13.743203 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:13.743208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:13.744370 (+ 1162us) server_negotiation.cc:407] Connection header received
0808 02:10:13.750024 (+ 5654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:13.750028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:13.750091 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:13.750158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:13.752088 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.752981 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.754952 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.755365 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.755457 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:13.760945 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:13.760951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:13.760954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:13.761001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:13.763152 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:13.764245 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:13.764249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:13.764252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:13.764353 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:13.765034 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:13.765038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:13.765040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:13.765307 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:13.765449 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:13.766281 (+ 832us) server_negotiation.cc:299] Negotiation successful
0808 02:10:13.766442 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:10:13.768878 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:13.741854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.66:33421)
0808 02:10:13.744162 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:13.744231 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:13.749716 (+ 5485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:13.750649 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:13.750660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:13.751094 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:13.751872 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:13.751896 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.753188 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.753192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:13.754789 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:13.754802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:13.755698 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:13.755703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:13.755933 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:13.756760 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:13.756784 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:13.760709 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:13.763662 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:13.763669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:13.763690 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:13.764103 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:13.764487 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:13.764491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:13.764493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:13.764705 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:13.765532 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:13.765540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:13.765928 (+ 388us) client_negotiation.cc:769] Sending connection context
0808 02:10:13.767217 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0808 02:10:13.767612 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":179,"threads_started":1}
I20260808 02:10:13.770736 22965 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:13.771018 22965 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:13.771663 22965 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:13.773032 22504 ts_manager.cc:194] Registered new tserver with Master: 9398a041c217426e86a2e1546bf8cbf2 (127.5.209.66:34063)
I20260808 02:10:13.773877 22504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:33421
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:10:13.817557 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:10:13 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
W20260808 02:10:13.980805 22971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:13.981194 22971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:13.981288 22971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:13.985574 22971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:13.985668 22971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:13.985790 22971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:13.991571 22971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:13.992807 22971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:13.993959 22971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:14.004317 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.004412 22977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.004509 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:14.005481 22971 server_base.cc:1034] running on GCE node
I20260808 02:10:14.005896 22971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:14.006587 22971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:14.007767 22971 hybrid_clock.cc:648] HybridClock initialized: now 1786155014007736 us; error 52 us; skew 500 ppm
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:14.010799 22971 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:14.011792 22971 webserver.cc:460] Webserver started at http://127.5.209.67:40565/ using document root <none> and password file <none>
I20260808 02:10:14.012497 22971 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:14.012591 22971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:14.012883 22971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:14.030081 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "d26d71e2d473441c87ef07ca7dae8ed4"
format_stamp: "Formatted at 2026-08-08 02:10:14 on dist-test-slave-bw88"
server_key: "4587a1418fc6d11b4b5b20a95b978c5d"
server_key_iv: "ee614633628557c85655cfe67d962405"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:14.035337 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "d26d71e2d473441c87ef07ca7dae8ed4"
format_stamp: "Formatted at 2026-08-08 02:10:14 on dist-test-slave-bw88"
server_key: "4587a1418fc6d11b4b5b20a95b978c5d"
server_key_iv: "ee614633628557c85655cfe67d962405"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:14.042877 22971 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 02:10:14.061725 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.063513 22971 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 02:10:14.063650 22971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "d26d71e2d473441c87ef07ca7dae8ed4"
format_stamp: "Formatted at 2026-08-08 02:10:14 on dist-test-slave-bw88"
server_key: "4587a1418fc6d11b4b5b20a95b978c5d"
server_key_iv: "ee614633628557c85655cfe67d962405"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:14.063788 22971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:14.088220 22971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:14.092121 22971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:14.092406 22971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:14.093259 22971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:14.095634 22971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:14.095750 22971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:10:14.095862 22971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:14.095942 22971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.107774 22971 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:46189
I20260808 02:10:14.107810 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:46189 every 8 connection(s)
I20260808 02:10:14.109150 22971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:14.110911 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 22971
I20260808 02:10:14.111081 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:10:14.111428 5957 external_mini_cluster.cc:1428] Setting key 6fad8b6ba5ecfb3161710a8371bda677
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:14.135735 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:14.113073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:37665 (local address 127.5.209.126:33321)
0808 02:10:14.113241 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:14.113246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:14.114912 (+ 1666us) server_negotiation.cc:407] Connection header received
0808 02:10:14.116183 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:14.116187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:14.116241 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:14.116293 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:14.118164 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.119019 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.120736 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.121050 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.121118 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:14.130412 (+ 9294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:14.130427 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:14.130430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:14.130456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:14.132004 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:14.132684 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:14.132688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:14.132689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:14.132788 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:14.134246 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:14.134270 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:14.134271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:14.134522 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:14.134617 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:14.135404 (+ 787us) server_negotiation.cc:299] Negotiation successful
0808 02:10:14.135555 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:10:14.136796 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:14.113523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.67:37665)
0808 02:10:14.114755 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:14.114814 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:14.115919 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:14.116729 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:14.116738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:14.117122 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:14.117952 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:14.117978 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.119187 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.119191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:14.120586 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:14.120598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.121364 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.121368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:14.121560 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:14.128559 (+ 6999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:14.128578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:14.130228 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:14.132136 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:14.132142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:14.132161 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:14.132553 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:14.132911 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:14.132914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:14.132916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:14.133124 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:14.134695 (+ 1571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:14.134702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:14.135089 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 02:10:14.135343 (+ 254us) client_negotiation.cc:240] Negotiation successful
0808 02:10:14.135692 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":142,"threads_started":1}
I20260808 02:10:14.138257 23130 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:14.138527 23130 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:14.139170 23130 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:14.140277 22504 ts_manager.cc:194] Registered new tserver with Master: d26d71e2d473441c87ef07ca7dae8ed4 (127.5.209.67:46189)
I20260808 02:10:14.141052 22504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:37665
I20260808 02:10:14.151348 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:10:14.152285 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 22612
I20260808 02:10:14.184837 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 22795
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155013, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:14.198447 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 22971
I20260808 02:10:14.210259 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 22463
I20260808 02:10:14.224305 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33321
--webserver_interface=127.5.209.126
--webserver_port=39751
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33321
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:14.357743 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:14.358006 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:14.358044 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:14.362033 23135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:14.362088 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:14.362107 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:14.362147 23135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:14.362233 23135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:14.367691 23135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33321
--ranger_config_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:14.368783 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:14.369915 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:14.378865 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.378922 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:14.379123 23135 server_base.cc:1034] running on GCE node
W20260808 02:10:14.379299 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:14.379909 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:14.381040 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:14.382246 23135 hybrid_clock.cc:648] HybridClock initialized: now 1786155014382204 us; error 63 us; skew 500 ppm
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:14.385182 23135 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:14.386155 23135 webserver.cc:460] Webserver started at http://127.5.209.126:39751/ using document root <none> and password file <none>
I20260808 02:10:14.386711 23135 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:14.386782 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:14.443090 23135 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260808 02:10:14.446349 23156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.448045 23135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:10:14.448190 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "05b5bccc57f64ce8bc5658fd064d2ee9"
format_stamp: "Formatted at 2026-08-08 02:10:12 on dist-test-slave-bw88"
server_key: "2a27043e36a0c1c390039bc0ba394d5c"
server_key_iv: "d40280ab1ff7a1932ba7e18360fbdc60"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:14.448925 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:14.478302 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:14.481468 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:14.481679 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:14.490298 23135 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33321
I20260808 02:10:14.490307 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33321 every 8 connection(s)
I20260808 02:10:14.491397 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:14.494522 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23135
I20260808 02:10:14.494987 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:41531
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:33321
--webserver_port=46153
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:14.497321 23226 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:10:14.499549 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Bootstrap starting.
I20260808 02:10:14.521777 23226 log.cc:824] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:14.530499 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:10:14.530978 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Bootstrap complete.
I20260808 02:10:14.536718 23226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:14.537256 23226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b5bccc57f64ce8bc5658fd064d2ee9, State: Initialized, Role: FOLLOWER
I20260808 02:10:14.537853 23226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:14.538007 23226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:14.538075 23226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:14.538164 23226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:10:14.540633 23226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:14.541010 23226 leader_election.cc:302] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b5bccc57f64ce8bc5658fd064d2ee9; no voters:
I20260808 02:10:14.541399 23226 leader_election.cc:288] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:10:14.541622 23233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:10:14.541889 23233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [term 2 LEADER]: Becoming Leader. State: Replica: 05b5bccc57f64ce8bc5658fd064d2ee9, State: Running, Role: LEADER
I20260808 02:10:14.542263 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } }
I20260808 02:10:14.542688 23226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:14.543377 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } } }
I20260808 02:10:14.543563 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:14.543826 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b5bccc57f64ce8bc5658fd064d2ee9. Latest consensus state: current_term: 2 leader_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b5bccc57f64ce8bc5658fd064d2ee9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33321 } } }
I20260808 02:10:14.543957 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:14.544266 23245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:14.547717 23245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:14.548236 23226 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:10:14.548518 23245 catalog_manager.cc:1259] Loaded cluster ID: 7da78fc08d29432ea9626306a4f8ea3f
I20260808 02:10:14.548619 23245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:14.551208 23245 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:14.551923 23245 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Loaded TSK: 0
I20260808 02:10:14.553501 23245 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:10:14.648156 23230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:14.648581 23230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:14.648667 23230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:14.654953 23230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:14.655045 23230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:14.655143 23230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:14.663676 23230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:41531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=46153
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:14.665128 23230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:14.666530 23230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:14.679953 23271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.680294 23272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.680923 23274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:14.681216 23230 server_base.cc:1034] running on GCE node
I20260808 02:10:14.681819 23230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:14.682721 23230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:14.686003 23230 hybrid_clock.cc:648] HybridClock initialized: now 1786155014685692 us; error 316 us; skew 500 ppm
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:14.690219 23230 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:14.691496 23230 webserver.cc:460] Webserver started at http://127.5.209.65:46153/ using document root <none> and password file <none>
I20260808 02:10:14.692351 23230 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:14.692535 23230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:14.727671 23230 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.003s sys 0.000s
I20260808 02:10:14.730652 23285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.732252 23230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:10:14.732383 23230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "75ca753344c1484fac1af38b8ded5935"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "5d0c57b15e5dbcee8c1d8d037311dd39"
server_key_iv: "87a79c6c4e93cdae3a72ad7607a7f792"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:14.733145 23230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:14.767084 23230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:14.771090 23230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:14.771314 23230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:14.772035 23230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:14.773186 23230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:14.773250 23230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.773305 23230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:14.773331 23230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:14.785315 23230 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:41531
I20260808 02:10:14.785375 23427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:41531 every 8 connection(s)
I20260808 02:10:14.786757 23230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:14.793143 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23230
I20260808 02:10:14.793706 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:34063
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:33321
--webserver_port=37911
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:14.810890 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:14.789271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.65:58443)
0808 02:10:14.789878 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:14.789930 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:14.790980 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:14.792605 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:14.792618 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:14.792951 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:14.793727 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:14.793750 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.795367 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.795370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:14.796815 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:14.796830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.797953 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.797957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:14.798149 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:14.798821 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:14.798844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:14.800659 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:14.807231 (+ 6572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:14.807238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:14.807254 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:14.807627 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:14.808161 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:14.808165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:14.808167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:14.808348 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:14.809255 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:14.809260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:14.809593 (+ 333us) client_negotiation.cc:769] Sending connection context
0808 02:10:14.809818 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 02:10:14.810089 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":128,"threads_started":1}
I20260808 02:10:14.814258 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:14.789148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:58443 (local address 127.5.209.126:33321)
0808 02:10:14.791503 (+ 2355us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:14.791525 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:14.791575 (+ 50us) server_negotiation.cc:407] Connection header received
0808 02:10:14.791853 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:14.791864 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:14.792159 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:14.792428 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:14.794057 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.795176 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.796968 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:14.797751 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:14.797956 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:14.800890 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:14.800897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:14.800916 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:14.800948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:14.807092 (+ 6144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:14.807783 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:14.807790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:14.807796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:14.807953 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:14.808466 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:14.808469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:14.808471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:14.808983 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:14.809153 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:14.812945 (+ 3792us) server_negotiation.cc:299] Negotiation successful
0808 02:10:14.813268 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2182,"thread_start_us":148,"threads_started":1}
I20260808 02:10:14.817163 23428 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:14.817457 23428 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:14.818102 23428 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:14.820449 23428 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:33321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:10:14.820587 23428 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:14.820785 23428 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:14.821739 23428 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:14.821919 23428 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:14.822666 23428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:10:14.950009 23436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:14.950413 23436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:14.950506 23436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:14.957141 23436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:14.957229 23436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:14.957322 23436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:14.964543 23436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:34063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=37911
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:14.965927 23436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:14.967443 23436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:14.981069 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.981065 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:14.981082 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:14.981524 23436 server_base.cc:1034] running on GCE node
I20260808 02:10:14.982048 23436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:14.982815 23436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:14.984108 23436 hybrid_clock.cc:648] HybridClock initialized: now 1786155014984085 us; error 38 us; skew 500 ppm
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:14.987535 23436 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:14.988641 23436 webserver.cc:460] Webserver started at http://127.5.209.66:37911/ using document root <none> and password file <none>
I20260808 02:10:14.989271 23436 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:14.989423 23436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:10:14 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:15.009126 23436 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260808 02:10:15.012424 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.014072 23436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:10:15.014189 23436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "9398a041c217426e86a2e1546bf8cbf2"
format_stamp: "Formatted at 2026-08-08 02:10:13 on dist-test-slave-bw88"
server_key: "427858add18119cd473d12bdfb6bd9cd"
server_key_iv: "06b69ba68d283ec3615162c36f5b48ce"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:15.014837 23436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:15.050196 23436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:15.054073 23436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:15.054383 23436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:15.055225 23436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:15.056466 23436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:15.056582 23436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.056704 23436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:15.056782 23436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.070093 23436 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:34063
I20260808 02:10:15.071382 23436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:15.072239 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:34063 every 8 connection(s)
I20260808 02:10:15.073978 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23436
I20260808 02:10:15.074522 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.209.84:59811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:46189
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:33321
--webserver_port=40565
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:10:15 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155014, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:15.098867 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:15.073546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:52359 (local address 127.5.209.126:33321)
0808 02:10:15.073704 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:15.073710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:15.075564 (+ 1854us) server_negotiation.cc:407] Connection header received
0808 02:10:15.077031 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:15.077035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:15.077104 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:15.077184 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:15.078760 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.079603 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.081599 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.082041 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.082156 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:15.091822 (+ 9666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:15.091829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:15.091834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:15.091870 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:15.095892 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.096683 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.096688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.096692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.096797 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.097437 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.097441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.097443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.097689 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:15.097773 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:15.098515 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 02:10:15.098665 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:10:15.100852 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:15.074058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.66:52359)
0808 02:10:15.075393 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:15.075454 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:15.076785 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:15.077436 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:15.077448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:15.077824 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:15.078552 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:15.078576 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.079787 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.079793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:15.081417 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:15.081433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.082413 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.082420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:15.082677 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:15.083410 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:15.083438 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:15.091237 (+ 7799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:15.096077 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:15.096086 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:15.096107 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:15.096538 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:15.096935 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:15.096941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:15.096945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:15.097187 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:15.097845 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:15.097852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:15.098220 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 02:10:15.099359 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0808 02:10:15.099731 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":170,"threads_started":1}
I20260808 02:10:15.102420 23609 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:15.102698 23609 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.103330 23609 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:15.105172 23609 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:33321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:10:15.105302 23609 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.105585 23609 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:15.106493 23609 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.106693 23609 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:15.107525 23609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:10:15.233965 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:15.234272 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:15.234352 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:15.238729 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:15.238811 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:15.238893 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:15.244673 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.209.84:59811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:46189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=40565
--tserver_master_addrs=127.5.209.126:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:15.245867 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:15.246977 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:15.258208 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:15.259039 23620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:15.259159 23623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:15.259675 23612 server_base.cc:1034] running on GCE node
I20260808 02:10:15.260226 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:15.261118 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:15.262363 23612 hybrid_clock.cc:648] HybridClock initialized: now 1786155015262338 us; error 62 us; skew 500 ppm
Aug 08 02:10:15 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:15.266161 23612 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:15.267284 23612 webserver.cc:460] Webserver started at http://127.5.209.67:40565/ using document root <none> and password file <none>
I20260808 02:10:15.268024 23612 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:15.268136 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:10:15 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:15.301743 23612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:10:15.304562 23632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.306037 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:10:15.306200 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "d26d71e2d473441c87ef07ca7dae8ed4"
format_stamp: "Formatted at 2026-08-08 02:10:14 on dist-test-slave-bw88"
server_key: "4587a1418fc6d11b4b5b20a95b978c5d"
server_key_iv: "ee614633628557c85655cfe67d962405"
server_key_version: "kuduclusterkey@0"
I20260808 02:10:15.306865 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:15.325152 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:15.328933 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:15.329236 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:15.330062 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:15.331223 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:15.331300 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.331362 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:15.331405 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:15.345546 23612 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:46189
I20260808 02:10:15.347131 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:15.349792 23775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:46189 every 8 connection(s)
I20260808 02:10:15.356222 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23612
Aug 08 02:10:15 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:15.363517 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:15.349688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:37285 (local address 127.5.209.126:33321)
0808 02:10:15.349837 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:15.349841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:15.351061 (+ 1220us) server_negotiation.cc:407] Connection header received
0808 02:10:15.352334 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:15.352338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:15.352395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:15.352449 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:15.354111 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.354725 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.355954 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.356205 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.356331 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:15.359247 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:15.359250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:15.359253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:15.359287 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:15.361082 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.361751 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.361755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.361757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.361852 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.362235 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.362238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.362240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.362427 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:15.362535 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:15.363147 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 02:10:15.363277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:10:15.365165 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:15.350328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33321 (local address 127.5.209.67:37285)
0808 02:10:15.350901 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:15.350946 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:15.352027 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:15.352981 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:15.352991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:15.353296 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:15.353953 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:15.353970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.354860 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.354863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:15.355792 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:15.355802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.356443 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.356449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:15.356624 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:15.357422 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:15.357446 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:15.359065 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:15.361248 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:15.361255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:15.361272 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:15.361635 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:15.361951 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:15.361955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:15.361957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:15.362138 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:15.362559 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:15.362566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:15.362835 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 02:10:15.363764 (+ 929us) client_negotiation.cc:240] Negotiation successful
0808 02:10:15.364118 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":146,"threads_started":1}
I20260808 02:10:15.368172 23776 heartbeater.cc:347] Connected to a master server at 127.5.209.126:33321
I20260808 02:10:15.368425 23776 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.369019 23776 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:15.370748 23776 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:33321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:10:15.370927 23776 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.371134 23776 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:15.372102 23776 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.372313 23776 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
W20260808 02:10:15.373263 23776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:10:15.384982 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:15.357500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.5.209.126:33321)
0808 02:10:15.358266 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:15.358278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:15.358293 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:10:15.358334 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:15.358337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:15.358390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:15.360419 (+ 2029us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:15.361454 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.363986 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.375696 (+ 11710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:15.375957 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:15.376103 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:15.378036 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:15.378040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:15.378043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:15.378073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:15.380106 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.380686 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.380690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.380693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.380797 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:15.381181 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:15.381184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:15.381186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:15.381480 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:15.381722 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:15.384692 (+ 2970us) server_negotiation.cc:299] Negotiation successful
0808 02:10:15.384823 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":107,"threads_started":1}
Aug 08 02:10:15 dist-test-slave-bw88 krb5kdc[21861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:15.822932 23428 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:15.823273 23428 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:15.824672 23178 ts_manager.cc:194] Registered new tserver with Master: 75ca753344c1484fac1af38b8ded5935 (127.5.209.65:41531)
I20260808 02:10:15.826524 23178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:58443
I20260808 02:10:16.107892 23609 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:16.108286 23609 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:16.109287 23178 ts_manager.cc:194] Registered new tserver with Master: 9398a041c217426e86a2e1546bf8cbf2 (127.5.209.66:34063)
I20260808 02:10:16.110173 23178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:52359
Aug 08 02:10:16 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
I20260808 02:10:16.373473 23776 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:16.373926 23776 heartbeater.cc:511] Master 127.5.209.126:33321 requested a full tablet report, sending...
I20260808 02:10:16.374770 23178 ts_manager.cc:194] Registered new tserver with Master: d26d71e2d473441c87ef07ca7dae8ed4 (127.5.209.67:46189)
I20260808 02:10:16.375707 23178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:37285
I20260808 02:10:16.376852 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:10:16.387204 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.378947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.5.209.126:33321)
0808 02:10:16.379199 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.379205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.379405 (+ 200us) server_negotiation.cc:407] Connection header received
0808 02:10:16.379552 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.379557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.379605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.379704 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:16.380401 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.380933 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.381879 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.382114 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.382226 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.383709 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:16.383713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:16.383715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:16.383745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:16.385321 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:16.385860 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:16.385864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:16.385866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:16.385956 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:16.386252 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:16.386256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:16.386258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:16.386437 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:16.386600 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.386866 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.386995 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260808 02:10:16.391042 23178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:10:16.394842 23178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:10:16.419461 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.407534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41531 (local address 127.0.0.1:54060)
0808 02:10:16.410547 (+ 3013us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.410628 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:16.410772 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:16.411796 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:16.411804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:16.411827 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:16.411941 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.411946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.413676 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.413680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.415041 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.415052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.416361 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.416364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.419000 (+ 2636us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.419059 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 02:10:16.419171 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:10:16.419299 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2798,"mutex_wait_us":4,"thread_start_us":124,"threads_started":1}
I20260808 02:10:16.420499 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.409384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.5.209.67:46189)
0808 02:10:16.413643 (+ 4259us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.413650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.413673 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:10:16.413746 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.413751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.413961 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.414113 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:16.414488 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.416352 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.417778 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.418577 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.418677 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.418821 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.420272 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.420369 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4138,"thread_start_us":101,"threads_started":1}
I20260808 02:10:16.420171 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.408782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46189 (local address 127.0.0.1:52658)
0808 02:10:16.412205 (+ 3423us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.412219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:16.413076 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:16.414265 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:16.414268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:16.414286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:16.414363 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.414369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.416469 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.416472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.417638 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.417648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.418882 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.418885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.418978 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.419820 (+ 842us) client_negotiation.cc:769] Sending connection context
0808 02:10:16.419893 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:10:16.419971 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3350,"mutex_wait_us":825,"thread_start_us":88,"threads_started":1}
I20260808 02:10:16.422521 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.409088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:34063 (local address 127.0.0.1:34172)
0808 02:10:16.413264 (+ 4176us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.413277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:16.413389 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:16.417125 (+ 3736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:16.417128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:16.417146 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:16.417246 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.417251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.418872 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.418876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.420786 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.420797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.422147 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.422151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.422247 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.422270 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:10:16.422317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 02:10:16.422377 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4113,"thread_start_us":103,"threads_started":1}
I20260808 02:10:16.423205 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.409748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.5.209.66:34063)
0808 02:10:16.416687 (+ 6939us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.416694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.416716 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:10:16.416787 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.416792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.416974 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.417403 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:16.417542 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.418744 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.420962 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.422008 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.422808 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.422941 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.422990 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.423074 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6813,"thread_start_us":133,"threads_started":1}
I20260808 02:10:16.424513 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.407899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.5.209.65:41531)
0808 02:10:16.411081 (+ 3182us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.411089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.411115 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:10:16.411196 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.411202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.411425 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.411598 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:16.412081 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.413543 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.415187 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.416012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.416126 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.416290 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.420883 (+ 4593us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.421018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3066,"thread_start_us":205,"threads_started":1}
I20260808 02:10:16.425180 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 22a880c75f2d4386b9e095277cd8c30c (DEFAULT_TABLE table=test-table [id=429c32e5a926445881bb46a73d31cd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:16.427423 23345 tablet_service.cc:1467] Processing CreateTablet for tablet 22a880c75f2d4386b9e095277cd8c30c (DEFAULT_TABLE table=test-table [id=429c32e5a926445881bb46a73d31cd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:16.427616 23516 tablet_service.cc:1467] Processing CreateTablet for tablet 22a880c75f2d4386b9e095277cd8c30c (DEFAULT_TABLE table=test-table [id=429c32e5a926445881bb46a73d31cd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:16.428624 23516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a880c75f2d4386b9e095277cd8c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:16.428624 23345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a880c75f2d4386b9e095277cd8c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:16.435161 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a880c75f2d4386b9e095277cd8c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:16.443289 23823 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Bootstrap starting.
I20260808 02:10:16.445962 23823 tablet_bootstrap.cc:654] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:16.445988 23824 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Bootstrap starting.
I20260808 02:10:16.446902 23823 log.cc:824] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:16.447669 23822 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Bootstrap starting.
I20260808 02:10:16.449334 23824 tablet_bootstrap.cc:654] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:16.449884 23822 tablet_bootstrap.cc:654] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:16.450378 23824 log.cc:824] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:16.450634 23822 log.cc:824] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:16.453585 23823 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: No bootstrap required, opened a new log
I20260808 02:10:16.453662 23822 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: No bootstrap required, opened a new log
I20260808 02:10:16.453794 23822 ts_tablet_manager.cc:1397] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:10:16.453794 23823 ts_tablet_manager.cc:1397] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 02:10:16.458455 23823 raft_consensus.cc:348] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.458856 23823 raft_consensus.cc:374] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:16.458959 23823 raft_consensus.cc:729] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9398a041c217426e86a2e1546bf8cbf2, State: Initialized, Role: FOLLOWER
I20260808 02:10:16.459080 23822 raft_consensus.cc:348] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.459431 23822 raft_consensus.cc:374] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:16.459525 23822 raft_consensus.cc:729] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ca753344c1484fac1af38b8ded5935, State: Initialized, Role: FOLLOWER
I20260808 02:10:16.459611 23823 consensus_queue.cc:260] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.460044 23822 consensus_queue.cc:260] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.460747 23823 ts_tablet_manager.cc:1428] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 02:10:16.461017 23822 ts_tablet_manager.cc:1428] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 02:10:16.461210 23609 heartbeater.cc:503] Master 127.5.209.126:33321 was elected leader, sending a full tablet report...
I20260808 02:10:16.461416 23428 heartbeater.cc:503] Master 127.5.209.126:33321 was elected leader, sending a full tablet report...
I20260808 02:10:16.464202 23824 tablet_bootstrap.cc:492] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: No bootstrap required, opened a new log
I20260808 02:10:16.464430 23824 ts_tablet_manager.cc:1397] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.005s
I20260808 02:10:16.468547 23824 raft_consensus.cc:348] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.468839 23824 raft_consensus.cc:374] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:16.468895 23824 raft_consensus.cc:729] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26d71e2d473441c87ef07ca7dae8ed4, State: Initialized, Role: FOLLOWER
I20260808 02:10:16.469589 23824 consensus_queue.cc:260] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.470722 23824 ts_tablet_manager.cc:1428] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:10:16.471139 23776 heartbeater.cc:503] Master 127.5.209.126:33321 was elected leader, sending a full tablet report...
Aug 08 02:10:16 dist-test-slave-bw88 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155015, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for HTTP/127.5.209.84@KRBTEST.COM
W20260808 02:10:16.539917 23429 tablet.cc:2369] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:10:16.574452 23610 tablet.cc:2369] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:10:16.601832 23777 tablet.cc:2369] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:10:16.625360 23832 raft_consensus.cc:482] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:16.625620 23832 raft_consensus.cc:504] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.627234 23832 leader_election.cc:288] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75ca753344c1484fac1af38b8ded5935 (127.5.209.65:41531), d26d71e2d473441c87ef07ca7dae8ed4 (127.5.209.67:46189)
I20260808 02:10:16.634583 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.628676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:37677 (local address 127.5.209.67:46189)
0808 02:10:16.628798 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.628802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.629072 (+ 270us) server_negotiation.cc:407] Connection header received
0808 02:10:16.629323 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.629327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.629415 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.629471 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:16.629953 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.630733 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.633189 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.633826 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.633929 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.633988 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.634360 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.634446 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:10:16.635270 23837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.628170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46189 (local address 127.5.209.66:37677)
0808 02:10:16.628978 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.628993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:16.629178 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:16.629640 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:16.629643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:16.629663 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:16.629807 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.629814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.630868 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.630871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.633038 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.633049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.634090 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.634094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.634200 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.634227 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:10:16.635016 (+ 789us) client_negotiation.cc:240] Negotiation successful
0808 02:10:16.635097 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":884,"thread_start_us":102,"threads_started":1}
I20260808 02:10:16.637387 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.627799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:35917 (local address 127.5.209.65:41531)
0808 02:10:16.627940 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.627944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.628629 (+ 685us) server_negotiation.cc:407] Connection header received
0808 02:10:16.630024 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.630027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.630075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.630122 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:16.630496 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.631197 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.635570 (+ 4373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.636192 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.636311 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.636361 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.637164 (+ 803us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.637252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:10:16.637421 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a880c75f2d4386b9e095277cd8c30c" candidate_uuid: "9398a041c217426e86a2e1546bf8cbf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" is_pre_election: true
I20260808 02:10:16.637794 23717 raft_consensus.cc:2454] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9398a041c217426e86a2e1546bf8cbf2 in term 0.
I20260808 02:10:16.638031 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.627711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41531 (local address 127.5.209.66:35917)
0808 02:10:16.628514 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.628532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:16.629883 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:16.630254 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:16.630257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:16.630273 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:16.630382 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.630387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.631343 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.631346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.632579 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:16.632591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.636494 (+ 3903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.636497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:16.636616 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.636639 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:10:16.637809 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0808 02:10:16.637894 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":30,"thread_start_us":119,"threads_started":1}
I20260808 02:10:16.638644 23460 leader_election.cc:302] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9398a041c217426e86a2e1546bf8cbf2, d26d71e2d473441c87ef07ca7dae8ed4; no voters:
I20260808 02:10:16.638775 23349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a880c75f2d4386b9e095277cd8c30c" candidate_uuid: "9398a041c217426e86a2e1546bf8cbf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ca753344c1484fac1af38b8ded5935" is_pre_election: true
I20260808 02:10:16.639009 23832 raft_consensus.cc:2781] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:10:16.639104 23832 raft_consensus.cc:482] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:10:16.639160 23832 raft_consensus.cc:3037] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:16.639115 23349 raft_consensus.cc:2454] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9398a041c217426e86a2e1546bf8cbf2 in term 0.
I20260808 02:10:16.640591 23832 raft_consensus.cc:504] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.640995 23832 leader_election.cc:288] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [CANDIDATE]: Term 1 election: Requested vote from peers 75ca753344c1484fac1af38b8ded5935 (127.5.209.65:41531), d26d71e2d473441c87ef07ca7dae8ed4 (127.5.209.67:46189)
I20260808 02:10:16.641666 23349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a880c75f2d4386b9e095277cd8c30c" candidate_uuid: "9398a041c217426e86a2e1546bf8cbf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ca753344c1484fac1af38b8ded5935"
I20260808 02:10:16.641790 23349 raft_consensus.cc:3037] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:16.643143 23349 raft_consensus.cc:2454] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9398a041c217426e86a2e1546bf8cbf2 in term 1.
I20260808 02:10:16.643873 23458 leader_election.cc:302] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ca753344c1484fac1af38b8ded5935, 9398a041c217426e86a2e1546bf8cbf2; no voters:
I20260808 02:10:16.644249 23832 raft_consensus.cc:2781] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:16.644434 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a880c75f2d4386b9e095277cd8c30c" candidate_uuid: "9398a041c217426e86a2e1546bf8cbf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26d71e2d473441c87ef07ca7dae8ed4"
I20260808 02:10:16.644484 23832 raft_consensus.cc:686] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 1 LEADER]: Becoming Leader. State: Replica: 9398a041c217426e86a2e1546bf8cbf2, State: Running, Role: LEADER
I20260808 02:10:16.644568 23717 raft_consensus.cc:3037] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:16.644867 23832 consensus_queue.cc:237] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } }
I20260808 02:10:16.645903 23717 raft_consensus.cc:2454] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9398a041c217426e86a2e1546bf8cbf2 in term 1.
I20260808 02:10:16.648792 23178 catalog_manager.cc:5495] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9398a041c217426e86a2e1546bf8cbf2 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "9398a041c217426e86a2e1546bf8cbf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9398a041c217426e86a2e1546bf8cbf2" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 34063 } health_report { overall_health: HEALTHY } } }
I20260808 02:10:16.717777 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.713413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.5.209.66:34063)
0808 02:10:16.713562 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:16.713566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:16.713585 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:10:16.713695 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:16.713698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:16.713773 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:16.713900 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:10:16.714146 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.714989 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.715893 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:16.716241 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:16.716356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:16.716683 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:10:16.717367 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:10:16.717500 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:16.717537 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 02:10:16.717596 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:10:16.724574 23717 raft_consensus.cc:1261] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 1 FOLLOWER]: Refusing update from remote peer 9398a041c217426e86a2e1546bf8cbf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:16.724575 23349 raft_consensus.cc:1261] T 22a880c75f2d4386b9e095277cd8c30c P 75ca753344c1484fac1af38b8ded5935 [term 1 FOLLOWER]: Refusing update from remote peer 9398a041c217426e86a2e1546bf8cbf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:16.725559 23842 consensus_queue.cc:1035] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75ca753344c1484fac1af38b8ded5935" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:16.725754 23832 consensus_queue.cc:1035] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26d71e2d473441c87ef07ca7dae8ed4" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:16.735659 23854 mvcc.cc:205] Tried to move back new op lower bound from 7316090948496437248 to 7316090948181360640. Current Snapshot: MvccSnapshot[applied={T|T < 7316090948496437248}]
I20260808 02:10:16.739003 23852 mvcc.cc:205] Tried to move back new op lower bound from 7316090948496437248 to 7316090948181360640. Current Snapshot: MvccSnapshot[applied={T|T < 7316090948496437248 or (T in {7316090948496437248})}]
I20260808 02:10:16.739291 23853 mvcc.cc:205] Tried to move back new op lower bound from 7316090948496437248 to 7316090948181360640. Current Snapshot: MvccSnapshot[applied={T|T < 7316090948496437248 or (T in {7316090948496437248})}]
I20260808 02:10:16.746521 23178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:16.746701 23178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:16.749949 23178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b5bccc57f64ce8bc5658fd064d2ee9: Sending DeleteTablet for 3 replicas of tablet 22a880c75f2d4386b9e095277cd8c30c
I20260808 02:10:16.750881 23516 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a880c75f2d4386b9e095277cd8c30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:16 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:34172
I20260808 02:10:16.750977 23687 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a880c75f2d4386b9e095277cd8c30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:16 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:52658
I20260808 02:10:16.751506 23863 tablet_replica.cc:330] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: stopping tablet replica
I20260808 02:10:16.751861 23863 raft_consensus.cc:2229] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:10:16.752146 23345 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a880c75f2d4386b9e095277cd8c30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:16 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:54060
I20260808 02:10:16.752334 23863 raft_consensus.cc:2258] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:16.753515 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23230
I20260808 02:10:16.757086 23863 ts_tablet_manager.cc:1905] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:16.759881 23863 ts_tablet_manager.cc:1918] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:16.760656 23864 tablet_replica.cc:330] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: stopping tablet replica
I20260808 02:10:16.761068 23864 raft_consensus.cc:2229] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:16.761255 23863 log.cc:1196] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/22a880c75f2d4386b9e095277cd8c30c
I20260808 02:10:16.761693 23863 ts_tablet_manager.cc:1939] T 22a880c75f2d4386b9e095277cd8c30c P 9398a041c217426e86a2e1546bf8cbf2: Deleting consensus metadata
I20260808 02:10:16.762115 23864 raft_consensus.cc:2258] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:16.763481 23159 catalog_manager.cc:4841] TS 9398a041c217426e86a2e1546bf8cbf2 (127.5.209.66:34063): tablet 22a880c75f2d4386b9e095277cd8c30c (table test-table [id=429c32e5a926445881bb46a73d31cd9c]) successfully deleted
I20260808 02:10:16.765391 23864 ts_tablet_manager.cc:1905] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:16.768239 23864 ts_tablet_manager.cc:1918] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:16.768400 23864 log.cc:1196] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/22a880c75f2d4386b9e095277cd8c30c
I20260808 02:10:16.768814 23864 ts_tablet_manager.cc:1939] T 22a880c75f2d4386b9e095277cd8c30c P d26d71e2d473441c87ef07ca7dae8ed4: Deleting consensus metadata
I20260808 02:10:16.770251 23160 catalog_manager.cc:4841] TS d26d71e2d473441c87ef07ca7dae8ed4 (127.5.209.67:46189): tablet 22a880c75f2d4386b9e095277cd8c30c (table test-table [id=429c32e5a926445881bb46a73d31cd9c]) successfully deleted
W20260808 02:10:16.772702 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:41531 (error 108)
I20260808 02:10:16.773939 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:16.773564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41531 (local address 127.0.0.1:54068)
0808 02:10:16.773764 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:16.773839 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260808 02:10:16.774137 23158 catalog_manager.cc:4584] TS 75ca753344c1484fac1af38b8ded5935 (127.5.209.65:41531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22a880c75f2d4386b9e095277cd8c30c: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41531: connect: Connection refused (error 111)
I20260808 02:10:16.774537 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23436
I20260808 02:10:16.787593 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23612
I20260808 02:10:16.801770 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23135
I20260808 02:10:16.814481 5957 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:10:17.159940 5957 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:10:17.160061 5957 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:10:17.505928 5957 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:10:17.506 UTC [21899] LOG: received smart shutdown request
2026-08-08 02:10:17.511 UTC [21899] LOG: background worker "logical replication launcher" (PID 21914) exited with exit code 1
2026-08-08 02:10:17.511 UTC [21907] LOG: shutting down
2026-08-08 02:10:17.674 UTC [21899] LOG: database system is shut down
2026-08-08T02:10:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51162 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:10:17 dist-test-slave-bw88 krb5kdc[23878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:10:17 dist-test-slave-bw88 krb5kdc[23878](info): set up 2 sockets
Aug 08 02:10:17 dist-test-slave-bw88 krb5kdc[23878](info): commencing operation
krb5kdc: starting...
W20260808 02:10:18.994606 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:10:19.137162 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33205
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:19.254020 23893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:19.254347 23893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:19.254449 23893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:19.258563 23893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:19.258644 23893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:19.258684 23893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:19.258742 23893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:19.258782 23893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:19.264010 23893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:19.265087 23893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:19.266141 23893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:19.274698 23900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:19.274686 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:19.274684 23901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:19.275010 23893 server_base.cc:1034] running on GCE node
I20260808 02:10:19.275714 23893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:19.276813 23893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:19.277980 23893 hybrid_clock.cc:648] HybridClock initialized: now 1786155019277944 us; error 46 us; skew 500 ppm
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:19.280776 23893 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:19.281685 23893 webserver.cc:460] Webserver started at http://127.5.209.126:36689/ using document root <none> and password file <none>
I20260808 02:10:19.282195 23893 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:19.282256 23893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:19.282420 23893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:19.284420 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "fd402f79903d48428895f26a5311fc63"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "34ae009104b779b088b83ae164a98b78"
server_key_iv: "67293a1ff92550e292a56b1f7c356213"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.284955 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "fd402f79903d48428895f26a5311fc63"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "34ae009104b779b088b83ae164a98b78"
server_key_iv: "67293a1ff92550e292a56b1f7c356213"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.288539 23893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:10:19.290957 23912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:19.292146 23893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:10:19.292285 23893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "fd402f79903d48428895f26a5311fc63"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "34ae009104b779b088b83ae164a98b78"
server_key_iv: "67293a1ff92550e292a56b1f7c356213"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.292393 23893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:19.306056 23893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:19.309381 23893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:19.309605 23893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:19.316831 23893 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33205
I20260808 02:10:19.316911 23981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33205 every 8 connection(s)
I20260808 02:10:19.317888 23893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:19.320681 23982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:19.324397 23982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.324916 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23893
I20260808 02:10:19.325052 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:10:19.325387 5957 external_mini_cluster.cc:1428] Setting key 1e842abb2e9d539aa29210cb4e83a152
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.125@KRBTEST.COM" created.
I20260808 02:10:19.339505 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.326980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33205 (local address 127.0.0.1:53602)
0808 02:10:19.327543 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.327607 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.328534 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.329200 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.329205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.329658 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.329907 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.329920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.331036 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.331040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.331763 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.331773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.332413 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.332417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.332508 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.333109 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.333126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.334617 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.336523 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.336534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.336544 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.336831 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.337171 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.337175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.337177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.337382 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.338155 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.338161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.338285 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.338434 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.338669 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260808 02:10:19.339510 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.327933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.5.209.126:33205)
0808 02:10:19.328255 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.328266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.328310 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:10:19.328833 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.328837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.329010 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.329202 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.330058 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.330908 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.331890 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.332262 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.332441 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.334759 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.334773 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.334783 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.334803 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.336413 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.336958 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.336966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.336968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.337071 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.337520 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.337524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.337526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.338015 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.338195 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.338585 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.338765 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
I20260808 02:10:19.342345 23982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.343117 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.342854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44714)
0808 02:10:19.342975 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.343044 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:10:19.343396 23916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
I20260808 02:10:19.344269 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.344043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44730)
0808 02:10:19.344168 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.344213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 02:10:19.344488 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:36081: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
I20260808 02:10:19.354557 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:36081
--webserver_interface=127.5.209.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:19.393738 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } attempt: 1
I20260808 02:10:19.394739 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.394413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44740)
0808 02:10:19.394583 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.394654 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 02:10:19.395555 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.395321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44756)
0808 02:10:19.395439 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.395495 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:10:19.395704 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:36081: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
I20260808 02:10:19.463897 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } attempt: 2
I20260808 02:10:19.464861 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.464517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44760)
0808 02:10:19.464709 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.464776 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 02:10:19.465764 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.465532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44774)
0808 02:10:19.465650 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.465704 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:10:19.465900 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:36081: Network error: Client connection negotiation failed: client connection to 127.5.209.125:36081: connect: Connection refused (error 111)
W20260808 02:10:19.474174 23990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:19.474435 23990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:19.474539 23990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:19.478504 23990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:19.478578 23990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:19.478626 23990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:19.478682 23990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:19.478719 23990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:19.483958 23990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:36081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.209.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:19.484997 23990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:19.486064 23990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:19.495075 23997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:19.495044 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:19.495108 23990 server_base.cc:1034] running on GCE node
W20260808 02:10:19.495038 23998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:19.495954 23990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:19.497223 23990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:19.498431 23990 hybrid_clock.cc:648] HybridClock initialized: now 1786155019498388 us; error 60 us; skew 500 ppm
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:19.501523 23990 init.cc:377] Logged in from keytab as kudu/127.5.209.125@KRBTEST.COM (short username kudu)
I20260808 02:10:19.502440 23990 webserver.cc:460] Webserver started at http://127.5.209.125:34517/ using document root <none> and password file <none>
I20260808 02:10:19.503048 23990 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:19.503149 23990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:19.503405 23990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:19.505606 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data/instance:
uuid: "843b35a9262e4ae3b4388b3c97390b8d"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "046d90eea9ae4ea3d137271c791105a8"
server_key_iv: "d6ed238a020b0c9b58776ef0bdd207fd"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.506214 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal/instance:
uuid: "843b35a9262e4ae3b4388b3c97390b8d"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "046d90eea9ae4ea3d137271c791105a8"
server_key_iv: "d6ed238a020b0c9b58776ef0bdd207fd"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.510262 23990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:10:19.512857 24010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:19.514174 23990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:10:19.514329 23990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "843b35a9262e4ae3b4388b3c97390b8d"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "046d90eea9ae4ea3d137271c791105a8"
server_key_iv: "d6ed238a020b0c9b58776ef0bdd207fd"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.514451 23990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:19.544315 23990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:19.548028 23990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:19.548265 23990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:19.556174 23990 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.125:36081
I20260808 02:10:19.556193 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.125:36081 every 8 connection(s)
I20260808 02:10:19.557093 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } attempt: 3
I20260808 02:10:19.557277 23990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
I20260808 02:10:19.560421 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:19.563185 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 23990
I20260808 02:10:19.563325 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-1/wal/instance
I20260808 02:10:19.563634 5957 external_mini_cluster.cc:1428] Setting key 2e47bac483846489fb1d0d36533b2f82
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:10:19.565670 24088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.570277 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.557642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44784)
0808 02:10:19.557788 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.557807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.557910 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.560186 (+ 2276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.560190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.560396 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.560557 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.560567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.561941 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.561945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.562749 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.562761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.563571 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.563574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.563689 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.564270 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.564292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.565629 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.568035 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.568038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.568040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.568298 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.568723 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.568725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.568727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.568808 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.569627 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.569633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.569768 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.569893 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.570065 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:10:19.572129 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.558373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.5.209.125:36081)
0808 02:10:19.558804 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.558827 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.558875 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:10:19.559525 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.559541 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.559898 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.560184 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.560705 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.561807 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.562876 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.563421 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.563615 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.565854 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.565859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.565872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.565900 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.567905 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.568427 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.568436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.568445 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.568590 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.568936 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.568939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.568941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.569489 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.569646 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.570620 (+ 974us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.570964 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":146,"threads_started":1}
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:19.574636 23982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.209.124@KRBTEST.COM; defaulting to no policy
I20260808 02:10:19.575448 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.575149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58598)
0808 02:10:19.575299 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.575365 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:10:19.576418 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.575988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58604)
0808 02:10:19.576285 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.576338 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
W20260808 02:10:19.576586 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.124@KRBTEST.COM" created.
I20260808 02:10:19.578214 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.567919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.5.209.126:33205)
0808 02:10:19.568051 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.568056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.568481 (+ 425us) server_negotiation.cc:407] Connection header received
0808 02:10:19.568693 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.568699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.568762 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.568852 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.569659 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.570762 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.571722 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.571981 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.572095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.574543 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.574546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.574549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.574573 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.576317 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.576911 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.576914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.576915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.576986 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.577246 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.577252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.577256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.577484 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.577564 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.577940 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.578044 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:10:19.578334 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.567874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33205 (local address 127.0.0.1:53604)
0808 02:10:19.568353 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.568417 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.568568 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.568915 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.568923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.569359 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.569520 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.569531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.570986 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.570989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.571596 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.571605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.572110 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.572114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.572248 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.572866 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.572878 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.574405 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.576441 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.576450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.576454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.576819 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.577071 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.577073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.577074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.577154 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.577567 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.577572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.577714 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.577837 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.578038 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260808 02:10:19.579638 24088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:10:19.588151 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.580229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44786)
0808 02:10:19.580343 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.580358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.580434 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.580761 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.580764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.580931 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.581013 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.581019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.581815 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.581820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.582343 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.582351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.582811 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.582814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.582902 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.583483 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.583495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.584958 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.586666 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.586671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.586673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.586868 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.587152 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.587155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.587157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.587240 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.587621 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.587625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.587716 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.587821 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.587966 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:10:19.588150 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.580493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.5.209.125:36081)
0808 02:10:19.580593 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.580596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.580605 (+ 9us) server_negotiation.cc:407] Connection header received
0808 02:10:19.580629 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.580631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.580661 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.580748 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.581127 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.581708 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.582482 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.582701 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.582810 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.585114 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.585117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.585119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.585140 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.586549 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.586967 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.586972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.586973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.587050 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.587333 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.587337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.587339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.587509 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.587621 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.587880 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.588025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":28}
I20260808 02:10:19.588925 24088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.589797 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.589537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58610)
0808 02:10:19.589662 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.589740 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:10:19.590472 24014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
I20260808 02:10:19.591435 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.591248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58626)
0808 02:10:19.591341 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.591384 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 02:10:19.591637 24088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
I20260808 02:10:19.594687 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:40407
--webserver_interface=127.5.209.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:19.607749 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 1
I20260808 02:10:19.608645 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.608334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58640)
0808 02:10:19.608498 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.608559 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 02:10:19.609505 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.609235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58642)
0808 02:10:19.609408 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.609453 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 02:10:19.609715 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
I20260808 02:10:19.640938 24088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 1
I20260808 02:10:19.641973 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.641588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58650)
0808 02:10:19.641789 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.641870 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 02:10:19.642773 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.642528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58660)
0808 02:10:19.642645 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.642701 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:10:19.642927 24088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
I20260808 02:10:19.684860 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 2
I20260808 02:10:19.685781 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.685478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58672)
0808 02:10:19.685637 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.685697 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:10:19.686533 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.686310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58678)
0808 02:10:19.686430 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.686479 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 02:10:19.686667 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
I20260808 02:10:19.711126 24088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 2
I20260808 02:10:19.712007 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.711644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58692)
0808 02:10:19.711826 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.711905 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:10:19.712913 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.712625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58700)
0808 02:10:19.712756 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.712822 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:10:19.713121 24088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
W20260808 02:10:19.715764 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:19.716037 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:19.716111 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:19.720319 24098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:19.720378 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:19.720398 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:19.720415 24098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:19.720430 24098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:19.725962 24098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33205,127.5.209.125:36081,127.5.209.124:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.209.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:19.727123 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:19.728207 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:19.737114 24114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:19.737149 24098 server_base.cc:1034] running on GCE node
W20260808 02:10:19.737079 24107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:19.737303 24111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:19.737888 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:19.738991 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:19.740159 24098 hybrid_clock.cc:648] HybridClock initialized: now 1786155019740144 us; error 27 us; skew 500 ppm
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:19.743165 24098 init.cc:377] Logged in from keytab as kudu/127.5.209.124@KRBTEST.COM (short username kudu)
I20260808 02:10:19.744046 24098 webserver.cc:460] Webserver started at http://127.5.209.124:33387/ using document root <none> and password file <none>
I20260808 02:10:19.744633 24098 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:19.744725 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:19.744971 24098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:19.747077 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data/instance:
uuid: "e6a5a0e7e8f7462fbde1280cdeee661e"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "1922da85cef1e7538ae95424e51022be"
server_key_iv: "b0c13e0a6c0758626ce21dc867a9d15b"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.747819 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal/instance:
uuid: "e6a5a0e7e8f7462fbde1280cdeee661e"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "1922da85cef1e7538ae95424e51022be"
server_key_iv: "b0c13e0a6c0758626ce21dc867a9d15b"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.751997 24098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 02:10:19.754626 24122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:19.755829 24098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:10:19.755972 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "e6a5a0e7e8f7462fbde1280cdeee661e"
format_stamp: "Formatted at 2026-08-08 02:10:19 on dist-test-slave-bw88"
server_key: "1922da85cef1e7538ae95424e51022be"
server_key_iv: "b0c13e0a6c0758626ce21dc867a9d15b"
server_key_version: "encryptionkey@0"
I20260808 02:10:19.756098 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:19.783320 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:19.785835 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 3
I20260808 02:10:19.786653 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.786355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58710)
0808 02:10:19.786508 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.786566 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:10:19.786685 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:19.786935 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:19.787451 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.787207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58714)
0808 02:10:19.787347 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.787393 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:10:19.787578 23982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:40407: Network error: Client connection negotiation failed: client connection to 127.5.209.124:40407: connect: Connection refused (error 111)
I20260808 02:10:19.794716 24098 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.124:40407
I20260808 02:10:19.794739 24192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.124:40407 every 8 connection(s)
I20260808 02:10:19.795704 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
I20260808 02:10:19.798869 24193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:19.802228 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24098
I20260808 02:10:19.802377 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786154850122309-5957-0/minicluster-data/master-2/wal/instance
I20260808 02:10:19.802678 5957 external_mini_cluster.cc:1428] Setting key 3308f0afe4dbcd79a0c37e0ecf3a0894
I20260808 02:10:19.803298 24193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.804296 24088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 3
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:19.817054 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.804746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58742)
0808 02:10:19.804927 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.804941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.805021 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.806877 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.806881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.807069 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.807192 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.807203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.808405 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.808409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.809219 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.809234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.809960 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.810058 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.810687 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.810707 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.812083 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.814886 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.814890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.814892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.815175 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.815578 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.815589 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.815682 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.816541 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.816543 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.816638 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.816734 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.816876 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 02:10:19.817144 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.805424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.5.209.126:33205)
0808 02:10:19.805585 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.805590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.805858 (+ 268us) server_negotiation.cc:407] Connection header received
0808 02:10:19.806275 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.806278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.806337 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.806411 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.807210 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.808020 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809511 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.809749 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809862 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.813486 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.813490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.813493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.813520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.815089 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815634 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.815637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.815639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.815735 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.816072 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.816076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.816078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.816285 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.816397 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.816823 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.816950 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:10:19.818279 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.804795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.5.209.124:40407)
0808 02:10:19.805160 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.805172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.805190 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:10:19.805686 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.805711 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.806079 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.806343 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.807304 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.808256 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809370 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.809771 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.810276 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.812246 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.812254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.812270 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.812302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.814766 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815290 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.815298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.815302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.815452 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815806 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.815811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.815815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.816357 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.816479 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.817377 (+ 898us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.817547 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":454,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":68608,"thread_start_us":87,"threads_started":1}
I20260808 02:10:19.818295 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.804129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.5.209.124:40407)
0808 02:10:19.804618 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.804641 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.804687 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:10:19.805685 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.805711 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.806079 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.807226 (+ 1147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.807327 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.808239 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809096 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.809728 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.809918 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.812889 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.812893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.812895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.812919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.815315 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815766 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.815770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.815772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.815855 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.816160 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.816165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.816167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.816369 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.816556 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.817433 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.817613 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260808 02:10:19.818280 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.805322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:33205 (local address 127.0.0.1:53612)
0808 02:10:19.805723 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.805770 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.806079 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.806471 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.806477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.806823 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.807066 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.807078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.808147 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.808150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.809394 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.809403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.810037 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.810041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.810188 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.810888 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.810913 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.813319 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.815199 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.815219 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.815515 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.815833 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.815836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.815838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.815957 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.816401 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.816410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.816556 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.816759 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.817426 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":26,"spinlock_wait_cycles":4480,"thread_start_us":116,"threads_started":1}
I20260808 02:10:19.819664 24193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:10:19.822757 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: Bootstrap starting.
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:10:19.825414 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:19.826222 24088 log.cc:824] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:19.828022 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.820284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.5.209.125:36081)
0808 02:10:19.820405 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.820410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.820426 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:10:19.820514 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.820517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.820569 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.820646 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.821117 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.821710 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.822473 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.822693 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.822808 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.824705 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.824713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.824715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.824736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.826201 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.826657 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.826660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.826662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.826749 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.827062 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.827066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.827067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.827352 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.827451 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.827780 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.827890 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:10:19.828022 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.820208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:36081 (local address 127.0.0.1:44788)
0808 02:10:19.820376 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.820387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.820446 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.820763 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.820765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.820913 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.820993 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.820998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.821835 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.821838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.822351 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.822360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.822813 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.822816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.822905 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.823365 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.823376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.824554 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.826320 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.826326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.826328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.826539 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.826883 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.826886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.826888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.826963 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.827502 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.827505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.827605 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.827723 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.827857 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 02:10:19.828987 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: No bootstrap required, opened a new log
I20260808 02:10:19.829100 24193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:19.833402 24088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.833678 24088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:19.833756 24088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843b35a9262e4ae3b4388b3c97390b8d, State: Initialized, Role: FOLLOWER
I20260808 02:10:19.834288 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
I20260808 02:10:19.834856 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.835469 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.835903 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:19.837878 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.827412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.5.209.126:33205)
0808 02:10:19.827535 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.827540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.827554 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:10:19.827600 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.827603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.827653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.827731 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.828777 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.829501 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.830556 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.830914 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.831016 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.833983 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.833987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.833990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.834019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.835897 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.836405 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.836410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.836412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.836507 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.836867 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.836870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.836873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.837113 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.837380 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.837552 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.837690 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:10:19.838393 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.829676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.5.209.124:40407)
0808 02:10:19.829768 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.829772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.829894 (+ 122us) server_negotiation.cc:407] Connection header received
0808 02:10:19.830001 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.830005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.830038 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.830129 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.830478 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.831203 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.832061 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.832302 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.832409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.834918 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.834922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.834924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.834946 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.836470 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.836993 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.836999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.837002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.837091 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.837444 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.837450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.837453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.837744 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.837843 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.838124 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.838243 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":34}
I20260808 02:10:19.838809 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.829641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58758)
0808 02:10:19.829825 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.829835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.829930 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.830153 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.830156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.830308 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.830367 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.830372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.831307 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.831313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.831931 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.831940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.832412 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.832415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.832511 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.833419 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.833436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.834766 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.836586 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.836592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.836594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.836873 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.837211 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.837214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.837215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.837308 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.837843 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.837846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.837941 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.838515 (+ 574us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.838658 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3}
W20260808 02:10:19.841660 24216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:10:19.841743 24216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:10:19.841998 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: Bootstrap starting.
I20260808 02:10:19.844702 24193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:19.845650 24193 log.cc:824] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:19.840629 23990 master_runner.cc:186] Error getting master registration for 127.5.209.126:33205: OK, instance_id { permanent_uuid: "fd402f79903d48428895f26a5311fc63" instance_seqno: 1786155019314923 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:10:19.846227 23990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:10:19.848138 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: No bootstrap required, opened a new log
I20260808 02:10:19.850947 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.839753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.5.209.125:36081)
0808 02:10:19.839858 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.839862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.841662 (+ 1800us) server_negotiation.cc:407] Connection header received
0808 02:10:19.841705 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.841708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.841757 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.841822 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.842570 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.843444 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.844535 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.844863 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.844966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.847726 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.847730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.847733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.847758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.849247 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.849710 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.849714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.849716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.849802 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.850096 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.850105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.850107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.850280 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.850485 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.850651 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.850770 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:10:19.852280 24193 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.852555 24193 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:19.852626 24193 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a5a0e7e8f7462fbde1280cdeee661e, State: Initialized, Role: FOLLOWER
I20260808 02:10:19.853111 24193 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.853649 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.853798 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.854628 24193 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:19.854867 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:10:19.858574 24098 master_runner.cc:186] Error getting master registration for 127.5.209.126:33205: OK, instance_id { permanent_uuid: "fd402f79903d48428895f26a5311fc63" instance_seqno: 1786155019314923 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:10:19.862069 24098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:10:19.868335 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.857471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.5.209.126:33205)
0808 02:10:19.857729 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.857733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.858110 (+ 377us) server_negotiation.cc:407] Connection header received
0808 02:10:19.858305 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.858309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.858356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.858416 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.859900 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.860618 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.862260 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.862590 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.862683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.864542 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.864546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.864548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.864574 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.866437 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.866925 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.866929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.866931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.867025 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.867347 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.867350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.867352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.867569 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.867686 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.868031 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.868167 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
W20260808 02:10:19.865307 24244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:10:19.870177 24244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:10:19.873081 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.857601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.5.209.124:40407)
0808 02:10:19.857831 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.857837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.858096 (+ 259us) server_negotiation.cc:407] Connection header received
0808 02:10:19.858305 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.858309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.858356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.858409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.859535 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.860391 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.861777 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.862094 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.862180 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.864443 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.864449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.864451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.864484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.869982 (+ 5498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.870620 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.870625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.870628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.870729 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.871104 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.871109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.871112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.871362 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.871522 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.872755 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.872907 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260808 02:10:19.875602 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.857530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.5.209.125:36081)
0808 02:10:19.857645 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.857652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.859131 (+ 1479us) server_negotiation.cc:407] Connection header received
0808 02:10:19.861633 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.861637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.861703 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.861761 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.866770 (+ 5009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.867527 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.869019 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.869317 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.869445 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.871575 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.871581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.871584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.871619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.873426 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.874167 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.874173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.874174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.874273 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.874611 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.874614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.874617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.874839 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.874949 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.875282 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.875404 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:10:19.925836 24219 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:19.926086 24219 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.927382 24219 leader_election.cc:288] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd402f79903d48428895f26a5311fc63 (127.5.209.126:33205), 843b35a9262e4ae3b4388b3c97390b8d (127.5.209.125:36081)
I20260808 02:10:19.928010 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843b35a9262e4ae3b4388b3c97390b8d" is_pre_election: true
I20260808 02:10:19.928493 24049 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6a5a0e7e8f7462fbde1280cdeee661e in term 0.
I20260808 02:10:19.928974 24126 leader_election.cc:302] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 843b35a9262e4ae3b4388b3c97390b8d, e6a5a0e7e8f7462fbde1280cdeee661e; no voters:
I20260808 02:10:19.929298 24219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:10:19.929419 24219 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:10:19.929486 24219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:19.930976 24219 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.931363 24219 leader_election.cc:288] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [CANDIDATE]: Term 1 election: Requested vote from peers fd402f79903d48428895f26a5311fc63 (127.5.209.126:33205), 843b35a9262e4ae3b4388b3c97390b8d (127.5.209.125:36081)
I20260808 02:10:19.931916 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843b35a9262e4ae3b4388b3c97390b8d"
I20260808 02:10:19.932080 24049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:19.933588 24049 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6a5a0e7e8f7462fbde1280cdeee661e in term 1.
I20260808 02:10:19.933938 24126 leader_election.cc:302] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 843b35a9262e4ae3b4388b3c97390b8d, e6a5a0e7e8f7462fbde1280cdeee661e; no voters:
I20260808 02:10:19.934995 24219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:19.935400 24219 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [term 1 LEADER]: Becoming Leader. State: Replica: e6a5a0e7e8f7462fbde1280cdeee661e, State: Running, Role: LEADER
I20260808 02:10:19.935963 24219 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.938499 24252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a5a0e7e8f7462fbde1280cdeee661e. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.938620 24252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:19.939061 24254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:19.940191 24254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:19.946874 24049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [term 1 FOLLOWER]: Refusing update from remote peer e6a5a0e7e8f7462fbde1280cdeee661e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:19.947621 24219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [LEADER]: Connected to new peer: Peer: permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:19.951738 23982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } attempt: 4
I20260808 02:10:19.952309 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a5a0e7e8f7462fbde1280cdeee661e. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.952443 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 02:10:19 dist-test-slave-bw88 krb5kdc[23878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155019, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
I20260808 02:10:19.961138 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.961293 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:19.961585 24255 mvcc.cc:205] Tried to move back new op lower bound from 7316090961691234304 to 7316090961661251584. Current Snapshot: MvccSnapshot[applied={T|T < 7316090961691234304}]
I20260808 02:10:19.963889 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.964006 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.966392 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.952719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.5.209.124:40407)
0808 02:10:19.952830 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:19.952835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:19.952848 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:10:19.952890 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:19.952893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:19.952942 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:19.953023 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:19.953695 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.954480 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.955652 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.955981 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.956082 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.959011 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:19.959014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:19.959017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:19.959044 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:19.962842 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.963437 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.963441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.963443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.963540 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:19.963972 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:19.963976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:19.963978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:19.964220 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:19.964316 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:19.965068 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 02:10:19.965717 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:10:19.966476 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:19.952209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:40407 (local address 127.0.0.1:58782)
0808 02:10:19.952470 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:19.952488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:19.952580 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:19.953162 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:19.953166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:19.953410 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:19.953546 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.953556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.954617 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.954621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.955472 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:19.955484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:19.956223 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:19.956227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:19.956319 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:19.956991 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:19.957007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:19.958796 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:19.963015 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.963018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.963020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.963276 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.963702 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:19.963705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:19.963707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:19.963804 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:19.964426 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:19.964429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:19.964642 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 02:10:19.965135 (+ 493us) client_negotiation.cc:240] Negotiation successful
0808 02:10:19.965675 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260808 02:10:19.969794 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: Bootstrap starting.
I20260808 02:10:19.972033 23982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:19.972822 23982 log.cc:824] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:19.973385 24254 catalog_manager.cc:1347] Generated new cluster ID: a0454b11d8a246448e3d220ab8fd144c
I20260808 02:10:19.973452 24254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:19.975211 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: No bootstrap required, opened a new log
I20260808 02:10:19.980494 23982 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.980815 23982 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:19.980929 23982 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd402f79903d48428895f26a5311fc63, State: Initialized, Role: FOLLOWER
I20260808 02:10:19.981546 23982 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } }
I20260808 02:10:19.983145 23982 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:19.983204 23947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd402f79903d48428895f26a5311fc63"
I20260808 02:10:19.983436 23947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:19.984961 23947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6a5a0e7e8f7462fbde1280cdeee661e in term 1.
I20260808 02:10:19.986418 23948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd402f79903d48428895f26a5311fc63" is_pre_election: true
I20260808 02:10:19.985980 23946 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [term 1 FOLLOWER]: Refusing update from remote peer e6a5a0e7e8f7462fbde1280cdeee661e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:19.987527 24219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:19.989464 24258 mvcc.cc:205] Tried to move back new op lower bound from 7316090961691234304 to 7316090961661251584. Current Snapshot: MvccSnapshot[applied={T|T < 7316090961691234304 or (T in {7316090961691234304})}]
I20260808 02:10:19.996467 24283 mvcc.cc:205] Tried to move back new op lower bound from 7316090961691234304 to 7316090961661251584. Current Snapshot: MvccSnapshot[applied={T|T < 7316090961691234304}]
I20260808 02:10:19.996703 24282 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.996858 24282 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.996966 24282 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a5a0e7e8f7462fbde1280cdeee661e. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.997049 24282 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.997133 24282 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:19.997198 24282 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:10:19.997989 24254 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:10:19.999238 24254 catalog_manager.cc:1504] Loading token signing keys...
W20260808 02:10:20.005141 24281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:10:20.005280 24281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:10:20.007057 24252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd402f79903d48428895f26a5311fc63" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33205 } } peers { permanent_uuid: "843b35a9262e4ae3b4388b3c97390b8d" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 36081 } } peers { permanent_uuid: "e6a5a0e7e8f7462fbde1280cdeee661e" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 40407 } } }
I20260808 02:10:20.007213 24252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:20.012378 24254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6a5a0e7e8f7462fbde1280cdeee661e: Generated new TSK 0
I20260808 02:10:20.013156 24254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:10:20.843181 24216 catalog_manager.cc:1259] Loaded cluster ID: a0454b11d8a246448e3d220ab8fd144c
I20260808 02:10:20.843277 24216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: loading cluster ID for follower catalog manager: success
I20260808 02:10:20.844934 24216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: acquiring CA information for follower catalog manager: success
I20260808 02:10:20.846102 24216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 843b35a9262e4ae3b4388b3c97390b8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:10:21.006664 24281 catalog_manager.cc:1259] Loaded cluster ID: a0454b11d8a246448e3d220ab8fd144c
I20260808 02:10:21.006752 24281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: loading cluster ID for follower catalog manager: success
I20260808 02:10:21.008529 24281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: acquiring CA information for follower catalog manager: success
I20260808 02:10:21.009832 24281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd402f79903d48428895f26a5311fc63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:10:21.092093 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23893
I20260808 02:10:21.104717 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 23990
I20260808 02:10:21.118266 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24098
2026-08-08T02:10:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3317 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:10:21 dist-test-slave-bw88 krb5kdc[24303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:10:21 dist-test-slave-bw88 krb5kdc[24303](info): set up 2 sockets
Aug 08 02:10:21 dist-test-slave-bw88 krb5kdc[24303](info): commencing operation
krb5kdc: starting...
W20260808 02:10:22.321667 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:10:22.452236 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33323
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:33323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:22.571411 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:22.571734 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:22.571816 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:22.575852 24320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:22.575937 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:22.575973 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:22.576030 24320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:22.576066 24320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:22.581218 24320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:33323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:33323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:22.582362 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:22.583458 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:10:22.592440 24320 server_base.cc:1034] running on GCE node
W20260808 02:10:22.592365 24336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:22.592355 24335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:22.592365 24338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:22.593182 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:22.594296 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:22.595492 24320 hybrid_clock.cc:648] HybridClock initialized: now 1786155022595472 us; error 36 us; skew 500 ppm
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155022, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:22.598563 24320 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:22.599538 24320 webserver.cc:460] Webserver started at http://127.5.209.126:42667/ using document root <none> and password file <none>
I20260808 02:10:22.600090 24320 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:22.600154 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:22.600327 24320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:22.602468 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "1a94b96b88c44ea69cc72f23f16dbc54"
format_stamp: "Formatted at 2026-08-08 02:10:22 on dist-test-slave-bw88"
server_key: "d741f4c0fa52af9c4b8f0254a18eb79a"
server_key_iv: "2f0825c9a56890a05da49cbe71ebb29f"
server_key_version: "encryptionkey@0"
I20260808 02:10:22.603075 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "1a94b96b88c44ea69cc72f23f16dbc54"
format_stamp: "Formatted at 2026-08-08 02:10:22 on dist-test-slave-bw88"
server_key: "d741f4c0fa52af9c4b8f0254a18eb79a"
server_key_iv: "2f0825c9a56890a05da49cbe71ebb29f"
server_key_version: "encryptionkey@0"
I20260808 02:10:22.607028 24320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:10:22.609503 24345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:22.610738 24320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:10:22.610841 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "1a94b96b88c44ea69cc72f23f16dbc54"
format_stamp: "Formatted at 2026-08-08 02:10:22 on dist-test-slave-bw88"
server_key: "d741f4c0fa52af9c4b8f0254a18eb79a"
server_key_iv: "2f0825c9a56890a05da49cbe71ebb29f"
server_key_version: "encryptionkey@0"
I20260808 02:10:22.610926 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:22.621084 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:22.624274 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:22.624424 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:22.631738 24320 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:33323
I20260808 02:10:22.631767 24413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:33323 every 8 connection(s)
I20260808 02:10:22.632676 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:22.635581 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:22.639191 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24320
I20260808 02:10:22.639323 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:10:22.639608 5957 external_mini_cluster.cc:1428] Setting key fd6bdeead07885b661a5287e8ba49db0
I20260808 02:10:22.641912 24414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54: Bootstrap starting.
I20260808 02:10:22.644767 24414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:22.645702 24414 log.cc:824] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:22.648378 24414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54: No bootstrap required, opened a new log
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:22.652976 24414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } }
I20260808 02:10:22.653301 24414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:22.653398 24414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a94b96b88c44ea69cc72f23f16dbc54, State: Initialized, Role: FOLLOWER
I20260808 02:10:22.653935 24414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } }
I20260808 02:10:22.654129 24414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:22.654219 24414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:22.654337 24414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:22.655311 24414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } }
I20260808 02:10:22.655576 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:22.640989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.5.209.126:33323)
0808 02:10:22.641533 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:22.641552 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:22.641610 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:10:22.642462 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:22.642487 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:22.642945 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:22.643333 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:22.644099 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:22.645674 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:22.646863 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:22.647541 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:22.647719 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:22.650558 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:22.650563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:22.650587 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:22.650618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:22.652386 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:22.652906 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:22.652911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:22.652916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:22.653023 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:22.653355 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:22.653358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:22.653360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:22.653818 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:22.653985 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:22.654444 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 02:10:22.654769 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":144,"threads_started":1}
I20260808 02:10:22.655701 24414 leader_election.cc:302] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a94b96b88c44ea69cc72f23f16dbc54; no voters:
I20260808 02:10:22.656060 24414 leader_election.cc:288] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:10:22.656193 24423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:22.656412 24423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [term 1 LEADER]: Becoming Leader. State: Replica: 1a94b96b88c44ea69cc72f23f16dbc54, State: Running, Role: LEADER
I20260808 02:10:22.656795 24423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } }
I20260808 02:10:22.657519 24414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:22.658573 24425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a94b96b88c44ea69cc72f23f16dbc54. Latest consensus state: current_term: 1 leader_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } } }
I20260808 02:10:22.658895 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:10:22.659071 24425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:22.658958 24424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a94b96b88c44ea69cc72f23f16dbc54" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 33323 } } }
I20260808 02:10:22.659313 24424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:22.660195 24436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:22.663038 24436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:22.668676 24436 catalog_manager.cc:1347] Generated new cluster ID: a220bf596ebe4c1b897efbea15f4cdca
I20260808 02:10:22.668748 24436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:22.673627 24436 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:10:22.674852 24436 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:22.683951 24436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a94b96b88c44ea69cc72f23f16dbc54: Generated new TSK 0
I20260808 02:10:22.684831 24436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:10:22.715503 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24320
2026-08-08T02:10:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1600 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24457](info): set up 2 sockets
Aug 08 02:10:22 dist-test-slave-bw88 krb5kdc[24457](info): commencing operation
krb5kdc: starting...
W20260808 02:10:23.901093 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:10:23 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:10:24.032275 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:24.152122 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:24.152416 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:24.152498 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:24.156482 24472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:24.156575 24472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:24.156615 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:24.156672 24472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:24.156711 24472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:24.161994 24472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:24.163069 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:24.164136 24472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:24.173106 24477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.173089 24478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.173153 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:24.173516 24472 server_base.cc:1034] running on GCE node
I20260808 02:10:24.174103 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:24.175169 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:24.176334 24472 hybrid_clock.cc:648] HybridClock initialized: now 1786155024176301 us; error 46 us; skew 500 ppm
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:24.185688 24472 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:24.186657 24472 webserver.cc:460] Webserver started at http://127.5.209.126:38469/ using document root <none> and password file <none>
I20260808 02:10:24.187255 24472 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:24.187318 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:24.187574 24472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:24.189775 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "451016ede2bc4dae96b4e689509441f3"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "2496a966f7552d674746fde94ff0fd0d"
server_key_iv: "e709602d585bee77c8e1713dd94695d9"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.190358 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "451016ede2bc4dae96b4e689509441f3"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "2496a966f7552d674746fde94ff0fd0d"
server_key_iv: "e709602d585bee77c8e1713dd94695d9"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.194080 24472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:10:24.196623 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.197911 24472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:10:24.198055 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "451016ede2bc4dae96b4e689509441f3"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "2496a966f7552d674746fde94ff0fd0d"
server_key_iv: "e709602d585bee77c8e1713dd94695d9"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.198170 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:24.239266 24472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:24.242508 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:24.242745 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:24.250128 24472 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34765
I20260808 02:10:24.250429 24554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34765 every 8 connection(s)
I20260808 02:10:24.251196 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:24.254272 24555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:24.259974 24555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap starting.
I20260808 02:10:24.260478 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24472
I20260808 02:10:24.260649 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:10:24.260923 5957 external_mini_cluster.cc:1428] Setting key 0ebc834cdd7f074d6d6cd7c365dad727
I20260808 02:10:24.262777 24555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:24.263530 24555 log.cc:824] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:24.265936 24555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: No bootstrap required, opened a new log
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:24.270591 24555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:24.270840 24555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:24.270927 24555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Initialized, Role: FOLLOWER
I20260808 02:10:24.271384 24555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:24.271543 24555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:24.271646 24555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:24.271775 24555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:24.272655 24555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:24.273016 24555 leader_election.cc:302] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451016ede2bc4dae96b4e689509441f3; no voters:
I20260808 02:10:24.273411 24555 leader_election.cc:288] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:10:24.273491 24562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:24.273707 24562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 LEADER]: Becoming Leader. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Running, Role: LEADER
I20260808 02:10:24.274062 24562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:24.274786 24555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:24.275298 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451016ede2bc4dae96b4e689509441f3. Latest consensus state: current_term: 1 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:24.275417 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:24.275790 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:24.275925 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:24.276696 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:24.262264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.5.209.126:34765)
0808 02:10:24.262678 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:24.262694 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:24.262729 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:10:24.263387 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:24.263405 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:24.263791 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:24.264158 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:24.264951 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.266102 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.267256 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.267782 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.267902 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:24.270962 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:24.270969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:24.270987 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:24.271042 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:24.273054 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.273625 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.273631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.273637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.273812 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.274121 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.274127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.274131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.274585 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:24.274808 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:24.275092 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:10:24.275705 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":142,"threads_started":1}
W20260808 02:10:24.280441 24587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:10:24.280524 24587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:10:24.280581 24570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:24.281538 24570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:24.286899 24570 catalog_manager.cc:1347] Generated new cluster ID: 436fcb16e4e34f05b9652531b4ed7d31
I20260808 02:10:24.286976 24570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:24.316934 24570 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:10:24.318019 24570 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:24.324049 24570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Generated new TSK 0
I20260808 02:10:24.324759 24570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:10:24.358577 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:24.500476 24596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:24.500720 24596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:24.500761 24596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:24.504815 24596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:24.504869 24596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:24.504976 24596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:24.510540 24596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:24.511606 24596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:24.512645 24596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:24.522496 24601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.522496 24602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.522516 24604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:24.522789 24596 server_base.cc:1034] running on GCE node
I20260808 02:10:24.523218 24596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:24.523921 24596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:24.525120 24596 hybrid_clock.cc:648] HybridClock initialized: now 1786155024525101 us; error 38 us; skew 500 ppm
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:24.527890 24596 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:24.528774 24596 webserver.cc:460] Webserver started at http://127.5.209.65:36139/ using document root <none> and password file <none>
I20260808 02:10:24.529428 24596 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:24.529526 24596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:24.529774 24596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:24.531831 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "cbdb64a1a3664acdbe998e0c8986e49d"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "d1372bcfc63094ebe2769b785e0b2449"
server_key_iv: "6d4aac41bdc6e9226756e621be212d53"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.532451 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "cbdb64a1a3664acdbe998e0c8986e49d"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "d1372bcfc63094ebe2769b785e0b2449"
server_key_iv: "6d4aac41bdc6e9226756e621be212d53"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.536239 24596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:10:24.538780 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.540028 24596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:10:24.540177 24596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "cbdb64a1a3664acdbe998e0c8986e49d"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "d1372bcfc63094ebe2769b785e0b2449"
server_key_iv: "6d4aac41bdc6e9226756e621be212d53"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.540297 24596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:24.554795 24596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:24.557746 24596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:24.557966 24596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:24.558610 24596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:24.559491 24596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:24.559566 24596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.559630 24596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:24.559681 24596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.569103 24596 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:39839
I20260808 02:10:24.569114 24767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:39839 every 8 connection(s)
I20260808 02:10:24.570183 24596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:24.576565 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24596
I20260808 02:10:24.576692 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:10:24.577018 5957 external_mini_cluster.cc:1428] Setting key fb1d01e5ec1abec1c85cb15274210e63
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:24.585228 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:24.572040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:45613 (local address 127.5.209.126:34765)
0808 02:10:24.572212 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:24.572217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:24.573555 (+ 1338us) server_negotiation.cc:407] Connection header received
0808 02:10:24.574465 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:24.574467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:24.574516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:24.574588 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:24.575974 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.576765 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.578236 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.578537 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.578658 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:24.581515 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:24.581518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:24.581520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:24.581545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:24.583007 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.583580 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.583584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.583587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.583669 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.584070 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.584073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.584075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.584273 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:24.584384 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:24.584910 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 02:10:24.585037 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:10:24.586087 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:24.572445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:45613)
0808 02:10:24.573394 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:24.573453 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:24.574241 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:24.574777 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:24.574784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:24.575327 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:24.575835 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:24.575849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.576897 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.576900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:24.578115 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:24.578126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.578737 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.578740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:24.578888 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:24.579507 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:24.579524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:24.581291 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:24.583172 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:24.583179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:24.583195 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:24.583465 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:24.583763 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:24.583767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:24.583769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:24.583963 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:24.584384 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:24.584392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:24.584663 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 02:10:24.584851 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 02:10:24.585193 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":568,"threads_started":1}
I20260808 02:10:24.587392 24768 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
I20260808 02:10:24.587692 24768 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:10:24.588330 24768 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:24.590009 24510 ts_manager.cc:194] Registered new tserver with Master: cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65:39839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:10:24.591251 24510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:45613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:10:24.607987 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:24.727632 24773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:24.727890 24773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:24.727957 24773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:24.731986 24773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:24.732043 24773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:24.732151 24773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:24.737450 24773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:24.738467 24773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:24.739512 24773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:24.749604 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.749675 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:24.749663 24773 server_base.cc:1034] running on GCE node
W20260808 02:10:24.749614 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:24.750197 24773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:24.750877 24773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:24.752067 24773 hybrid_clock.cc:648] HybridClock initialized: now 1786155024752052 us; error 27 us; skew 500 ppm
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:24.754868 24773 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:24.755719 24773 webserver.cc:460] Webserver started at http://127.5.209.66:41615/ using document root <none> and password file <none>
I20260808 02:10:24.756294 24773 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:24.756386 24773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:24.756631 24773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:24.758713 24773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "da85cc8a2911479c9d19d4a7e7026323"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "8fb148b2b93f7c206f66790c56f3a39a"
server_key_iv: "f8905917a792f659920cf13747eb18d7"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.759285 24773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "da85cc8a2911479c9d19d4a7e7026323"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "8fb148b2b93f7c206f66790c56f3a39a"
server_key_iv: "f8905917a792f659920cf13747eb18d7"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.762970 24773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 02:10:24.765599 24794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.766921 24773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:10:24.767041 24773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "da85cc8a2911479c9d19d4a7e7026323"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "8fb148b2b93f7c206f66790c56f3a39a"
server_key_iv: "f8905917a792f659920cf13747eb18d7"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.767150 24773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:24.788311 24773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:24.791193 24773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:24.791400 24773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:24.792024 24773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:24.792876 24773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:24.792925 24773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.792994 24773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:24.793030 24773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.802165 24773 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:33031
I20260808 02:10:24.802244 24950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:33031 every 8 connection(s)
I20260808 02:10:24.803190 24773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:24.805056 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24773
I20260808 02:10:24.805198 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:10:24.805527 5957 external_mini_cluster.cc:1428] Setting key a59b62989315560a454c53267cd989b0
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:10:24.817747 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:24.805091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:56549 (local address 127.5.209.126:34765)
0808 02:10:24.806031 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:24.806038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:24.806460 (+ 422us) server_negotiation.cc:407] Connection header received
0808 02:10:24.807333 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:24.807336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:24.807387 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:24.807467 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:24.808746 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.809583 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.810754 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.811170 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.811299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:24.813862 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:24.813869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:24.813875 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:24.813908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:24.815550 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.816065 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.816070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.816072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.816172 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:24.816517 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:24.816523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:24.816527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:24.816795 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:24.816905 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:24.817517 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 02:10:24.817624 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831}
I20260808 02:10:24.818666 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:24.805541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:56549)
0808 02:10:24.806300 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:24.806353 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:24.807130 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:24.807640 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:24.807647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:24.808050 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:24.808588 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:24.808611 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.809722 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.809725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:24.810628 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:24.810639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:24.811327 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:24.811330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:24.811472 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:24.812111 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:24.812128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:24.813682 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:24.815683 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:24.815688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:24.815700 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:24.815957 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:24.816276 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:24.816279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:24.816280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:24.816411 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:24.816909 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:24.816914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:24.817185 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 02:10:24.817461 (+ 276us) client_negotiation.cc:240] Negotiation successful
0808 02:10:24.817724 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":146,"threads_started":1}
I20260808 02:10:24.820003 24952 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:10:24.820262 24952 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:24.820770 24952 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:24.821969 24510 ts_manager.cc:194] Registered new tserver with Master: da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031)
I20260808 02:10:24.822558 24510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:56549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:10:24.836683 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:24.955241 24957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:24.955499 24957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:24.955543 24957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:24.959481 24957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:24.959538 24957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:24.959597 24957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:24.964835 24957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:24.965894 24957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:24.966859 24957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:24.976742 24967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.976823 24964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:24.976751 24962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:24.977008 24957 server_base.cc:1034] running on GCE node
I20260808 02:10:24.977465 24957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:24.978111 24957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:24.979282 24957 hybrid_clock.cc:648] HybridClock initialized: now 1786155024979267 us; error 26 us; skew 500 ppm
Aug 08 02:10:24 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:24.982045 24957 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:24.982867 24957 webserver.cc:460] Webserver started at http://127.5.209.67:34041/ using document root <none> and password file <none>
I20260808 02:10:24.983526 24957 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:24.983590 24957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:24.983801 24957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:24.985863 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "17bf9c770f574b4fbd39f042bba8a563"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "4fe555cb6c92d9429626a157b3eef133"
server_key_iv: "ab9ae360499a8d4bb83f7bfeb31c1ef8"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.986433 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "17bf9c770f574b4fbd39f042bba8a563"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "4fe555cb6c92d9429626a157b3eef133"
server_key_iv: "ab9ae360499a8d4bb83f7bfeb31c1ef8"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.990084 24957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:10:24.992506 24977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:24.993829 24957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:10:24.993963 24957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "17bf9c770f574b4fbd39f042bba8a563"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "4fe555cb6c92d9429626a157b3eef133"
server_key_iv: "ab9ae360499a8d4bb83f7bfeb31c1ef8"
server_key_version: "encryptionkey@0"
I20260808 02:10:24.994073 24957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:25.015416 24957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:25.018605 24957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:25.018889 24957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:25.019593 24957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:25.020550 24957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:25.020632 24957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.020714 24957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:25.020774 24957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.031160 24957 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:46093
I20260808 02:10:25.031198 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:46093 every 8 connection(s)
I20260808 02:10:25.032171 24957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:25.034332 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 24957
I20260808 02:10:25.034451 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:10:25.034773 5957 external_mini_cluster.cc:1428] Setting key 65cf7fe146b8f368bc0c8b7d99c4db19
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:25.046248 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.034182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:48687 (local address 127.5.209.126:34765)
0808 02:10:25.034355 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.034358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.035445 (+ 1087us) server_negotiation.cc:407] Connection header received
0808 02:10:25.036966 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.036970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.037028 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.037137 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:25.038433 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.038951 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.040062 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.040275 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.040352 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.042763 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:25.042766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:25.042768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:25.042792 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:25.044378 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.044832 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.044835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.044836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.044911 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.045244 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.045247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.045248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.045479 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:25.045561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.046015 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.046125 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:10:25.047049 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.034675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:48687)
0808 02:10:25.035266 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.035324 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.036727 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.037317 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.037323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.037762 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:25.038288 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.038309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.039060 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.039063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.039953 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.039962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.040402 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.040404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.040553 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.041080 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:25.041095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:25.042596 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:25.044479 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.044483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.044494 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.044745 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.045027 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.045029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.045031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.045160 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.045571 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:25.045576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:25.045807 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.045963 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.046209 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":162,"threads_started":1}
I20260808 02:10:25.048368 25130 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
I20260808 02:10:25.048635 25130 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:25.049226 25130 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:25.050292 24510 ts_manager.cc:194] Registered new tserver with Master: 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
I20260808 02:10:25.050879 24510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:48687
I20260808 02:10:25.060500 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:10:25.070112 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.062600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.5.209.126:34765)
0808 02:10:25.062716 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.062722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.062915 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:10:25.063088 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.063091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.063140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.063224 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:25.063969 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.064486 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.065389 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.065616 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.065731 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.067051 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:25.067055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:25.067058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:25.067092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:25.068276 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.068724 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.068728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.068730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.068816 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.069129 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.069133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.069134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.069429 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:25.069576 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.069825 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.069945 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 02:10:25.072860 24510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:10:25.075950 24510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:10:25.092350 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.084814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39839 (local address 127.0.0.1:48242)
0808 02:10:25.085864 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.085936 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.086130 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.087632 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.087635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.087654 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.087815 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.087823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.089672 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.089678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.090814 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.090826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.091777 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.091781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.091893 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.091929 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.092038 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.092170 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":67,"threads_started":1}
I20260808 02:10:25.092728 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.084886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.5.209.67:46093)
0808 02:10:25.085390 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.085404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.086021 (+ 617us) server_negotiation.cc:407] Connection header received
0808 02:10:25.086341 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.086349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.086588 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.086734 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.087746 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.089156 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.090890 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.091782 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.091878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.092023 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.092306 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.092542 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":183,"threads_started":1}
I20260808 02:10:25.092957 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.086122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.5.209.65:39839)
0808 02:10:25.086706 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.086725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.086750 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:10:25.087227 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.087233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.087484 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.087651 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.087976 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.089523 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.090967 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.091646 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.091776 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.091933 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.092692 (+ 759us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.092802 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":137,"threads_started":1}
I20260808 02:10:25.093879 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.085378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46093 (local address 127.0.0.1:38840)
0808 02:10:25.085864 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.085921 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.086135 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.087000 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.087008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.087038 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.087575 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.087588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.089363 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.089369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.090733 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.090747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.092055 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.092058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.092136 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.092156 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.092931 (+ 775us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.093025 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":4,"spinlock_wait_cycles":17280,"thread_start_us":132,"threads_started":1}
I20260808 02:10:25.096143 24680 tablet_service.cc:1467] Processing CreateTablet for tablet 3096265f651741bda96257913aa4b77e (DEFAULT_TABLE table=test-table [id=06d06b86c7f34d088d7314e98e702e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:25.096546 25047 tablet_service.cc:1467] Processing CreateTablet for tablet 3096265f651741bda96257913aa4b77e (DEFAULT_TABLE table=test-table [id=06d06b86c7f34d088d7314e98e702e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:25.097182 24680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3096265f651741bda96257913aa4b77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:25.097321 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.085794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:33031 (local address 127.0.0.1:55352)
0808 02:10:25.090848 (+ 5054us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.090864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.090947 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.093192 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.093197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.093216 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.093295 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.093299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.094734 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.094740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.095828 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.095839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.096940 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.096943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.097038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.097063 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.097130 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.097196 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4979,"thread_start_us":92,"threads_started":1}
I20260808 02:10:25.097455 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.089627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.5.209.66:33031)
0808 02:10:25.092511 (+ 2884us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.092524 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.092548 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:10:25.092626 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.092632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.092829 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.092993 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.093445 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.094571 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.095978 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.096790 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.096909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.097123 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.097179 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.097265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2712,"thread_start_us":934,"threads_started":1}
I20260808 02:10:25.097716 25047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3096265f651741bda96257913aa4b77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:25.099720 24855 tablet_service.cc:1467] Processing CreateTablet for tablet 3096265f651741bda96257913aa4b77e (DEFAULT_TABLE table=test-table [id=06d06b86c7f34d088d7314e98e702e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:25.100651 24855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3096265f651741bda96257913aa4b77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:25.103991 25156 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap starting.
I20260808 02:10:25.106213 25156 tablet_bootstrap.cc:654] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:25.106963 25156 log.cc:824] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:25.107532 25158 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Bootstrap starting.
I20260808 02:10:25.107741 25157 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Bootstrap starting.
I20260808 02:10:25.110301 25156 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: No bootstrap required, opened a new log
I20260808 02:10:25.110430 25157 tablet_bootstrap.cc:654] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:25.110423 25158 tablet_bootstrap.cc:654] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:25.110548 25156 ts_tablet_manager.cc:1397] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 02:10:25.111284 25158 log.cc:824] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:25.111284 25157 log.cc:824] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:25.113375 25157 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: No bootstrap required, opened a new log
I20260808 02:10:25.113560 25157 ts_tablet_manager.cc:1397] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:10:25.113828 25158 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: No bootstrap required, opened a new log
I20260808 02:10:25.114002 25158 ts_tablet_manager.cc:1397] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 02:10:25.114707 25156 raft_consensus.cc:348] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.115293 25156 raft_consensus.cc:374] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:25.115374 25156 raft_consensus.cc:729] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Initialized, Role: FOLLOWER
I20260808 02:10:25.116098 25156 consensus_queue.cc:260] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.116758 24768 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:25.117129 25156 ts_tablet_manager.cc:1428] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 02:10:25.117422 25157 raft_consensus.cc:348] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.117689 25157 raft_consensus.cc:374] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:25.117738 25157 raft_consensus.cc:729] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da85cc8a2911479c9d19d4a7e7026323, State: Initialized, Role: FOLLOWER
I20260808 02:10:25.117765 25158 raft_consensus.cc:348] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.118008 25158 raft_consensus.cc:374] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:25.118067 25158 raft_consensus.cc:729] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bf9c770f574b4fbd39f042bba8a563, State: Initialized, Role: FOLLOWER
I20260808 02:10:25.118167 25157 consensus_queue.cc:260] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.118458 25158 consensus_queue.cc:260] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.118870 24952 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:25.119093 25157 ts_tablet_manager.cc:1428] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 02:10:25.119344 25158 ts_tablet_manager.cc:1428] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 02:10:25.119678 25130 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:25.145324 25163 raft_consensus.cc:482] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:25.145622 25163 raft_consensus.cc:504] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.146978 25163 leader_election.cc:288] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093), da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031)
I20260808 02:10:25.151075 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.147293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:59651 (local address 127.5.209.67:46093)
0808 02:10:25.147422 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.147428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.147708 (+ 280us) server_negotiation.cc:407] Connection header received
0808 02:10:25.147878 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.147882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.147926 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.148011 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.148330 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.148844 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150064 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.150471 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150608 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.150667 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.150863 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.150938 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:10:25.151103 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.147400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:55491 (local address 127.5.209.66:33031)
0808 02:10:25.147554 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.147559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.147778 (+ 219us) server_negotiation.cc:407] Connection header received
0808 02:10:25.147931 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.147936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.147985 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.148069 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.148374 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.148868 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150076 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.150474 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.150671 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.150900 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.150972 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:10:25.151151 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.147177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46093 (local address 127.5.209.65:59651)
0808 02:10:25.147584 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.147598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.147749 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.148026 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.148030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.148046 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.148212 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.148218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.148968 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.148972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.149935 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.149946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.150596 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.150703 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.150755 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.150843 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.150917 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":35,"thread_start_us":135,"threads_started":1}
I20260808 02:10:25.151160 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.147268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:33031 (local address 127.5.209.65:55491)
0808 02:10:25.147666 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.147678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.147809 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.148099 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.148101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.148118 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.148253 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.148258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.148985 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.148990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.149937 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.149947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.150602 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.150606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.150766 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.150786 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.150879 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.150951 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":102,"thread_start_us":131,"threads_started":1}
I20260808 02:10:25.151988 24883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3096265f651741bda96257913aa4b77e" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da85cc8a2911479c9d19d4a7e7026323" is_pre_election: true
I20260808 02:10:25.151988 25072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3096265f651741bda96257913aa4b77e" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bf9c770f574b4fbd39f042bba8a563" is_pre_election: true
I20260808 02:10:25.152336 25072 raft_consensus.cc:2454] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 0.
I20260808 02:10:25.152335 24883 raft_consensus.cc:2454] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 0.
I20260808 02:10:25.152922 24619 leader_election.cc:302] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17bf9c770f574b4fbd39f042bba8a563, cbdb64a1a3664acdbe998e0c8986e49d, da85cc8a2911479c9d19d4a7e7026323; no voters:
I20260808 02:10:25.153239 25163 raft_consensus.cc:2781] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:10:25.153339 25163 raft_consensus.cc:482] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:10:25.153424 25163 raft_consensus.cc:3037] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:25.154531 25163 raft_consensus.cc:504] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.154897 25163 leader_election.cc:288] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 election: Requested vote from peers 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093), da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031)
I20260808 02:10:25.155265 24883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3096265f651741bda96257913aa4b77e" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da85cc8a2911479c9d19d4a7e7026323"
I20260808 02:10:25.155294 25072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3096265f651741bda96257913aa4b77e" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bf9c770f574b4fbd39f042bba8a563"
I20260808 02:10:25.155428 24883 raft_consensus.cc:3037] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:25.155439 25072 raft_consensus.cc:3037] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:25.156654 24883 raft_consensus.cc:2454] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 1.
I20260808 02:10:25.156689 25072 raft_consensus.cc:2454] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 1.
I20260808 02:10:25.156973 24625 leader_election.cc:302] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d, da85cc8a2911479c9d19d4a7e7026323; no voters:
I20260808 02:10:25.157166 25163 raft_consensus.cc:2781] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:25.157436 25163 raft_consensus.cc:686] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 LEADER]: Becoming Leader. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Running, Role: LEADER
I20260808 02:10:25.157820 25163 consensus_queue.cc:237] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.161108 24505 catalog_manager.cc:5495] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d reported cstate change: term changed from 0 to 1, leader changed from <none> to cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65). New cstate: current_term: 1 leader_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } health_report { overall_health: UNKNOWN } } }
I20260808 02:10:25.191252 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.187308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.5.209.65:39839)
0808 02:10:25.187462 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.187467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.187484 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:10:25.187694 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.187697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.187761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.187887 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:10:25.188243 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.188764 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.189727 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.189949 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.190065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.190397 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:10:25.190846 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:10:25.191000 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.191039 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.191101 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:10:25.197754 24883 raft_consensus.cc:1261] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Refusing update from remote peer cbdb64a1a3664acdbe998e0c8986e49d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:25.197754 25072 raft_consensus.cc:1261] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Refusing update from remote peer cbdb64a1a3664acdbe998e0c8986e49d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:25.198625 25173 consensus_queue.cc:1035] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:25.198899 25163 consensus_queue.cc:1035] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:25.208230 25178 mvcc.cc:205] Tried to move back new op lower bound from 7316090983201320960 to 7316090983049027584. Current Snapshot: MvccSnapshot[applied={T|T < 7316090983201320960}]
I20260808 02:10:25.211397 25177 mvcc.cc:205] Tried to move back new op lower bound from 7316090983201320960 to 7316090983049027584. Current Snapshot: MvccSnapshot[applied={T|T < 7316090983201320960 or (T in {7316090983201320960})}]
I20260808 02:10:25.215919 24505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:25.216099 24505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:25.218426 24505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Sending DeleteTablet for 3 replicas of tablet 3096265f651741bda96257913aa4b77e
I20260808 02:10:25.219206 25047 tablet_service.cc:1514] Processing DeleteTablet for tablet 3096265f651741bda96257913aa4b77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:38840
I20260808 02:10:25.219237 24680 tablet_service.cc:1514] Processing DeleteTablet for tablet 3096265f651741bda96257913aa4b77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48242
I20260808 02:10:25.219480 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24596
I20260808 02:10:25.219758 24855 tablet_service.cc:1514] Processing DeleteTablet for tablet 3096265f651741bda96257913aa4b77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:55352
I20260808 02:10:25.221912 25192 tablet_replica.cc:330] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: stopping tablet replica
I20260808 02:10:25.222054 25191 tablet_replica.cc:330] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: stopping tablet replica
I20260808 02:10:25.222247 25192 raft_consensus.cc:2229] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:25.222275 25191 raft_consensus.cc:2229] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:25.222570 25192 raft_consensus.cc:2258] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:25.222570 25191 raft_consensus.cc:2258] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:25.226008 25192 ts_tablet_manager.cc:1905] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:25.226289 25191 ts_tablet_manager.cc:1905] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:25.228323 25192 ts_tablet_manager.cc:1918] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:25.228439 25192 log.cc:1196] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/3096265f651741bda96257913aa4b77e
I20260808 02:10:25.228752 25192 ts_tablet_manager.cc:1939] T 3096265f651741bda96257913aa4b77e P da85cc8a2911479c9d19d4a7e7026323: Deleting consensus metadata
I20260808 02:10:25.229857 25191 ts_tablet_manager.cc:1918] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:25.229970 25191 log.cc:1196] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/3096265f651741bda96257913aa4b77e
I20260808 02:10:25.230047 24492 catalog_manager.cc:4841] TS da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): tablet 3096265f651741bda96257913aa4b77e (table test-table [id=06d06b86c7f34d088d7314e98e702e39]) successfully deleted
I20260808 02:10:25.230324 25191 ts_tablet_manager.cc:1939] T 3096265f651741bda96257913aa4b77e P 17bf9c770f574b4fbd39f042bba8a563: Deleting consensus metadata
I20260808 02:10:25.231668 24490 catalog_manager.cc:4841] TS 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): tablet 3096265f651741bda96257913aa4b77e (table test-table [id=06d06b86c7f34d088d7314e98e702e39]) successfully deleted
W20260808 02:10:25.234299 24491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:39839 (error 108)
I20260808 02:10:25.235136 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24773
I20260808 02:10:25.235476 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.235096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39839 (local address 127.0.0.1:48266)
0808 02:10:25.235300 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.235377 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 02:10:25.246640 24491 catalog_manager.cc:4584] TS cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65:39839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3096265f651741bda96257913aa4b77e: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39839: connect: Connection refused (error 111)
I20260808 02:10:25.256448 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24957
I20260808 02:10:25.278304 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 24472
I20260808 02:10:25.288425 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--webserver_interface=127.5.209.126
--webserver_port=38469
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:25.405797 25194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:25.406049 25194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:25.406121 25194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:25.410211 25194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:25.410265 25194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:25.410284 25194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:25.410337 25194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:25.410365 25194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:25.415693 25194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=38469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:25.416720 25194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:25.417780 25194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:25.432516 25205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:25.432586 25194 server_base.cc:1034] running on GCE node
W20260808 02:10:25.432551 25201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:25.432538 25209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:25.433422 25194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:25.434576 25194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:25.435789 25194 hybrid_clock.cc:648] HybridClock initialized: now 1786155025435761 us; error 41 us; skew 500 ppm
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:25.439049 25194 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:25.440008 25194 webserver.cc:460] Webserver started at http://127.5.209.126:38469/ using document root <none> and password file <none>
I20260808 02:10:25.440618 25194 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:25.440701 25194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:25.445096 25194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:10:25.447327 25218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.448588 25194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 02:10:25.448740 25194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "451016ede2bc4dae96b4e689509441f3"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "2496a966f7552d674746fde94ff0fd0d"
server_key_iv: "e709602d585bee77c8e1713dd94695d9"
server_key_version: "encryptionkey@0"
I20260808 02:10:25.449447 25194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:25.460247 25194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:25.463207 25194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:25.463415 25194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:25.470924 25194 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34765
I20260808 02:10:25.471217 25281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34765 every 8 connection(s)
I20260808 02:10:25.472076 25194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:25.475906 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25194
I20260808 02:10:25.476372 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:39839
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=36139
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:25.476704 25282 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:10:25.478403 25282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap starting.
I20260808 02:10:25.503651 25282 log.cc:824] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:25.516083 25282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:10:25.516553 25282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap complete.
I20260808 02:10:25.522336 25282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:25.522857 25282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Initialized, Role: FOLLOWER
I20260808 02:10:25.523449 25282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:25.523648 25282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:25.523729 25282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:25.523842 25282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:10:25.526505 25282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:25.526907 25282 leader_election.cc:302] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451016ede2bc4dae96b4e689509441f3; no voters:
I20260808 02:10:25.527300 25282 leader_election.cc:288] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:10:25.527441 25289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:10:25.527709 25289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 LEADER]: Becoming Leader. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Running, Role: LEADER
I20260808 02:10:25.528036 25289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:25.528649 25282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:25.529078 25292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451016ede2bc4dae96b4e689509441f3. Latest consensus state: current_term: 2 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:25.529225 25292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:25.529541 25290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:25.529649 25290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:25.531585 25294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:25.536857 25294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:25.537545 25294 catalog_manager.cc:1259] Loaded cluster ID: 436fcb16e4e34f05b9652531b4ed7d31
I20260808 02:10:25.537613 25294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:25.539065 25294 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 02:10:25.543967 25294 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 02:10:25.549693 25294 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:25.550496 25294 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Loaded TSK: 0
I20260808 02:10:25.551756 25294 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:10:25.614955 25286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:25.615232 25286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:25.615306 25286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:25.619292 25286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:25.619374 25286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:25.619464 25286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:25.624709 25286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:39839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=36139
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:25.625821 25286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:25.626860 25286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:25.638319 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:25.638402 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:25.638332 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:25.638788 25286 server_base.cc:1034] running on GCE node
I20260808 02:10:25.639214 25286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:25.639887 25286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:25.641091 25286 hybrid_clock.cc:648] HybridClock initialized: now 1786155025641077 us; error 32 us; skew 500 ppm
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:25.644335 25286 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:25.645423 25286 webserver.cc:460] Webserver started at http://127.5.209.65:36139/ using document root <none> and password file <none>
I20260808 02:10:25.646034 25286 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:25.646114 25286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:25.650184 25286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:10:25.652592 25336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.654115 25286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:10:25.654261 25286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "cbdb64a1a3664acdbe998e0c8986e49d"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "d1372bcfc63094ebe2769b785e0b2449"
server_key_iv: "6d4aac41bdc6e9226756e621be212d53"
server_key_version: "encryptionkey@0"
I20260808 02:10:25.654891 25286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:25.671463 25286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:25.675072 25286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:25.675293 25286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:25.675999 25286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:25.677248 25345 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:10:25.681619 25286 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:10:25.681687 25286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260808 02:10:25.681763 25286 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:10:25.683521 25286 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:10:25.683579 25286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 02:10:25.683877 25345 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap starting.
I20260808 02:10:25.695417 25286 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:39839
I20260808 02:10:25.695452 25480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:39839 every 8 connection(s)
I20260808 02:10:25.696775 25286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:25.705554 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25286
I20260808 02:10:25.706074 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:33031
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=41615
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:25.706969 25345 log.cc:824] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:25.714807 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.699236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:46847)
0808 02:10:25.699814 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.699859 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.700719 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.701994 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.702002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.702462 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:25.702948 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.702962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.704340 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.704343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.705249 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.705260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.706391 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.706395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.706560 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.707231 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:25.707249 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:25.709036 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:25.711453 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.711461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.711512 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.711793 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.712184 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.712188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.712190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.712384 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.712999 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:25.713005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:25.713266 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.713496 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.713863 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":164,"threads_started":1}
I20260808 02:10:25.714794 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.699164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:46847 (local address 127.5.209.126:34765)
0808 02:10:25.699574 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.699599 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.699992 (+ 393us) server_negotiation.cc:407] Connection header received
0808 02:10:25.701171 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.701183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.701590 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.701933 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:25.703148 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.704141 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.705456 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.706018 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.706233 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.709250 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:25.709256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:25.709287 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:25.709317 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:25.711208 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.711931 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.711937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.711946 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.712068 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.712490 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.712495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.712497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.712870 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:25.713027 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.713632 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.713902 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
I20260808 02:10:25.717603 25481 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
I20260808 02:10:25.717864 25481 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:25.718375 25481 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:25.718835 25345 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:10:25.719355 25345 tablet_bootstrap.cc:492] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap complete.
I20260808 02:10:25.720903 25241 ts_manager.cc:194] Registered new tserver with Master: cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65:39839)
I20260808 02:10:25.721294 25345 ts_tablet_manager.cc:1397] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.007s
I20260808 02:10:25.724524 25241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:46847
I20260808 02:10:25.724886 25345 raft_consensus.cc:348] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.725536 25345 raft_consensus.cc:729] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Initialized, Role: FOLLOWER
I20260808 02:10:25.726096 25345 consensus_queue.cc:260] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } } peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } }
I20260808 02:10:25.727197 25345 ts_tablet_manager.cc:1428] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:10:25.727385 25481 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:25.732067 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.724580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.5.209.65:39839)
0808 02:10:25.725902 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.725913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.725936 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:10:25.726002 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.726007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.726504 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.726652 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:25.727845 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.728883 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.730817 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.731409 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.731501 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.731610 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.731890 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.731965 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":117,"threads_started":1}
I20260808 02:10:25.732136 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.723843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39839 (local address 127.0.0.1:48270)
0808 02:10:25.725162 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.725207 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.725360 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.727536 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.727543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.727571 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:25.727704 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.727711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.729004 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.729008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.730675 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.730693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.731537 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.731542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.731678 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.731723 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.731843 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.731982 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":107,"threads_started":1}
I20260808 02:10:25.733312 25398 tablet_service.cc:1514] Processing DeleteTablet for tablet 3096265f651741bda96257913aa4b77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:25 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48270
I20260808 02:10:25.734176 25491 tablet_replica.cc:330] stopping tablet replica
I20260808 02:10:25.734478 25491 raft_consensus.cc:2229] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:25.734730 25491 raft_consensus.cc:2258] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:25.735702 25491 ts_tablet_manager.cc:1905] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:25.739254 25491 ts_tablet_manager.cc:1918] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:25.739341 25491 log.cc:1196] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/3096265f651741bda96257913aa4b77e
I20260808 02:10:25.739660 25491 ts_tablet_manager.cc:1939] T 3096265f651741bda96257913aa4b77e P cbdb64a1a3664acdbe998e0c8986e49d: Deleting consensus metadata
I20260808 02:10:25.741242 25222 catalog_manager.cc:4841] TS cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65:39839): tablet 3096265f651741bda96257913aa4b77e (table test-table [id=06d06b86c7f34d088d7314e98e702e39]) successfully deleted
W20260808 02:10:25.842171 25487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:25.842450 25487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:25.842540 25487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:25.847203 25487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:25.847263 25487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:25.847326 25487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:25.853148 25487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:33031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=41615
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:25.854368 25487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:25.855497 25487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:25.866289 25496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:25.866294 25502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:25.866349 25487 server_base.cc:1034] running on GCE node
W20260808 02:10:25.866278 25498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:25.866878 25487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:25.867591 25487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:25.868788 25487 hybrid_clock.cc:648] HybridClock initialized: now 1786155025868768 us; error 31 us; skew 500 ppm
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:25.871728 25487 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:25.872579 25487 webserver.cc:460] Webserver started at http://127.5.209.66:41615/ using document root <none> and password file <none>
I20260808 02:10:25.873145 25487 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:25.873243 25487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:25.877008 25487 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:10:25.879086 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.880286 25487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 02:10:25.880421 25487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "da85cc8a2911479c9d19d4a7e7026323"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "8fb148b2b93f7c206f66790c56f3a39a"
server_key_iv: "f8905917a792f659920cf13747eb18d7"
server_key_version: "encryptionkey@0"
I20260808 02:10:25.880959 25487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:25.914955 25487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:25.918435 25487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:25.918659 25487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:25.919322 25487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:25.920192 25487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:25.920240 25487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.920311 25487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:25.920356 25487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:25.929878 25487 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:33031
I20260808 02:10:25.930004 25662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:33031 every 8 connection(s)
I20260808 02:10:25.931005 25487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:25.936075 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25487
I20260808 02:10:25.936533 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:46093
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=34041
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:10:25 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:25.945045 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.932910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:33245 (local address 127.5.209.126:34765)
0808 02:10:25.933066 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:25.933071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:25.933991 (+ 920us) server_negotiation.cc:407] Connection header received
0808 02:10:25.935308 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:25.935311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:25.935368 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:25.935456 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:25.936757 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.937309 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.938517 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.938769 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.938890 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.941404 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:25.941407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:25.941409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:25.941434 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:25.942906 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.943447 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.943451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.943453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.943538 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:25.943909 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:25.943912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:25.943914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:25.944113 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:25.944223 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:25.944726 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 02:10:25.944864 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:10:25.945956 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:25.933361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:33245)
0808 02:10:25.933824 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:25.933869 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:25.935078 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:25.935648 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:25.935655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:25.936003 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:25.936563 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.936577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.937482 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.937485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.938380 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:25.938390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:25.938933 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:25.938937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:25.939118 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:25.939753 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:25.939770 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:25.941197 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:25.943049 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.943055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.943070 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.943336 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.943640 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:25.943643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:25.943646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:25.943792 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:25.944224 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:25.944231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:25.944481 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 02:10:25.944684 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 02:10:25.945014 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260808 02:10:25.947227 25663 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
I20260808 02:10:25.947497 25663 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:25.948028 25663 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:25.949240 25241 ts_manager.cc:194] Registered new tserver with Master: da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031)
I20260808 02:10:25.950047 25241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:33245
W20260808 02:10:26.056804 25668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:26.057063 25668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:26.057107 25668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:26.061285 25668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:26.061394 25668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:26.061463 25668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:26.066735 25668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:46093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=34041
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:26.067710 25668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:26.068670 25668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:26.079311 25674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.079311 25675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.079684 25679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:26.079471 25668 server_base.cc:1034] running on GCE node
I20260808 02:10:26.080188 25668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:26.080862 25668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:26.082120 25668 hybrid_clock.cc:648] HybridClock initialized: now 1786155026082084 us; error 44 us; skew 500 ppm
Aug 08 02:10:26 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:26.085290 25668 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:26.086323 25668 webserver.cc:460] Webserver started at http://127.5.209.67:34041/ using document root <none> and password file <none>
I20260808 02:10:26.087136 25668 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:26.087275 25668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:26.091683 25668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:10:26.093854 25688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:26.095090 25668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:10:26.095235 25668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "17bf9c770f574b4fbd39f042bba8a563"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "4fe555cb6c92d9429626a157b3eef133"
server_key_iv: "ab9ae360499a8d4bb83f7bfeb31c1ef8"
server_key_version: "encryptionkey@0"
I20260808 02:10:26.095790 25668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:26.108114 25668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:26.111263 25668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:26.111475 25668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:26.112139 25668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:26.113051 25668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:26.113126 25668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:26.113200 25668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:26.113243 25668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:26.123142 25668 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:46093
I20260808 02:10:26.123178 25847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:46093 every 8 connection(s)
I20260808 02:10:26.124190 25668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:26.124806 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25668
I20260808 02:10:26.136111 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.125687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.5.209.126:34765)
0808 02:10:26.125855 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.125860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.126276 (+ 416us) server_negotiation.cc:407] Connection header received
0808 02:10:26.126569 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.126572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.126637 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.126750 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:26.127741 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.128459 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.130026 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.130285 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.130370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.132232 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:26.132237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:26.132241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:26.132277 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:26.134171 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:26.134663 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:26.134667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:26.134670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:26.134784 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:26.135094 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:26.135101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:26.135106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:26.135393 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:26.135656 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.135757 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.135891 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 02:10:26 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:26.141006 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.126215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:49649 (local address 127.5.209.126:34765)
0808 02:10:26.126497 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.126505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.127959 (+ 1454us) server_negotiation.cc:407] Connection header received
0808 02:10:26.129232 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.129235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.129278 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.129420 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:26.131208 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.132063 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.134091 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.134403 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.134521 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.137742 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:26.137746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:26.137749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:26.137776 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:26.139214 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:26.139715 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:26.139717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:26.139718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:26.139794 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:26.140099 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:26.140101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:26.140102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:26.140259 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:26.140335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.140792 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.140888 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260808 02:10:26.141883 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.126728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:49649)
0808 02:10:26.127782 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.127842 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.128998 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.129640 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:26.129651 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:26.130236 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:26.131033 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.131061 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.132218 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.132222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.133936 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.133955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.134571 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.134574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.134838 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.135481 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:26.135508 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:26.137573 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:26.139340 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:26.139345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:26.139356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:26.139622 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:26.139878 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:26.139880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:26.139881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:26.140014 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:26.140350 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:26.140355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:26.140575 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 02:10:26.140741 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 02:10:26.140980 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":555,"threads_started":1}
I20260808 02:10:26.142999 25848 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34765
I20260808 02:10:26.143255 25848 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:26.143755 25848 heartbeater.cc:511] Master 127.5.209.126:34765 requested a full tablet report, sending...
I20260808 02:10:26.144755 25241 ts_manager.cc:194] Registered new tserver with Master: 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
I20260808 02:10:26.145314 25241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:49649
I20260808 02:10:26.150048 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:10:26.154564 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.150652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.5.209.126:34765)
0808 02:10:26.150773 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.150784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.150988 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:10:26.151107 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.151109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.151200 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.151282 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:10:26.151482 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.151960 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.153025 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.153281 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.153497 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.153757 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:10:26.154216 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:10:26.154327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.154393 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.154455 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:10:26.155251 25241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:10:26.159186 25241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:10:26.167867 25398 tablet_service.cc:1467] Processing CreateTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 (DEFAULT_TABLE table=test-table [id=cff9cbc67f21488481ce7c1680ee747c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:26.168646 25398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56cd2e6b38c44ed980051bfb7d351c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:26.171630 25345 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap starting.
I20260808 02:10:26.172559 25345 tablet_bootstrap.cc:654] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:26.172608 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.167205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46093 (local address 127.0.0.1:38856)
0808 02:10:26.167585 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.167598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.167696 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.168468 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:26.168471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:26.168498 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:26.168599 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.168606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.170141 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.170144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.171071 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.171079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.172254 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.172257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.172373 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.172393 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:10:26.172436 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 02:10:26.172482 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260808 02:10:26.172614 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.166720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:33031 (local address 127.0.0.1:55354)
0808 02:10:26.166844 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.166861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.166941 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.167884 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:26.167887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:26.167907 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:26.168011 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.168018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.169688 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.169693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.170684 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.170695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.172131 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.172135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.172243 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.172307 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 02:10:26.172372 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 02:10:26.172455 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":24}
I20260808 02:10:26.172708 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.166942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.5.209.66:33031)
0808 02:10:26.167333 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.167347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.167371 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:10:26.167456 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.167464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.167710 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.168318 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:26.168502 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.169561 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.170824 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.171998 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.172116 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.172266 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.172425 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.172557 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":188,"threads_started":1}
I20260808 02:10:26.173930 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.167586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.5.209.67:46093)
0808 02:10:26.167986 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.167997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.168019 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:10:26.168103 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.168108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.168309 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.168465 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:26.168751 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.169983 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.171241 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.172125 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.173490 (+ 1365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.173646 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.173702 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.173794 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":176,"threads_started":1}
I20260808 02:10:26.174271 25345 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: No bootstrap required, opened a new log
I20260808 02:10:26.174378 25345 ts_tablet_manager.cc:1397] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 02:10:26.174779 25345 raft_consensus.cc:348] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.174906 25345 raft_consensus.cc:374] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:26.174947 25345 raft_consensus.cc:729] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Initialized, Role: FOLLOWER
I20260808 02:10:26.175086 25345 consensus_queue.cc:260] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.175346 25345 ts_tablet_manager.cc:1428] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 02:10:26.176129 25576 tablet_service.cc:1467] Processing CreateTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 (DEFAULT_TABLE table=test-table [id=cff9cbc67f21488481ce7c1680ee747c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:26.176754 25759 tablet_service.cc:1467] Processing CreateTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 (DEFAULT_TABLE table=test-table [id=cff9cbc67f21488481ce7c1680ee747c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:26.177404 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56cd2e6b38c44ed980051bfb7d351c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:26.177879 25759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e56cd2e6b38c44ed980051bfb7d351c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:26.184448 25862 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Bootstrap starting.
I20260808 02:10:26.185496 25865 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Bootstrap starting.
I20260808 02:10:26.186717 25862 tablet_bootstrap.cc:654] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:26.187561 25865 tablet_bootstrap.cc:654] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:26.187845 25862 log.cc:824] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:26.188282 25865 log.cc:824] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:26.189949 25862 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: No bootstrap required, opened a new log
I20260808 02:10:26.190176 25862 ts_tablet_manager.cc:1397] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260808 02:10:26.190313 25865 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: No bootstrap required, opened a new log
I20260808 02:10:26.190496 25865 ts_tablet_manager.cc:1397] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 02:10:26.193156 25862 raft_consensus.cc:348] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.193444 25862 raft_consensus.cc:374] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:26.193514 25862 raft_consensus.cc:729] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bf9c770f574b4fbd39f042bba8a563, State: Initialized, Role: FOLLOWER
I20260808 02:10:26.193288 25865 raft_consensus.cc:348] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.193609 25865 raft_consensus.cc:374] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:26.193691 25865 raft_consensus.cc:729] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da85cc8a2911479c9d19d4a7e7026323, State: Initialized, Role: FOLLOWER
I20260808 02:10:26.193984 25862 consensus_queue.cc:260] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.194116 25865 consensus_queue.cc:260] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.194654 25848 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:26.194779 25663 heartbeater.cc:503] Master 127.5.209.126:34765 was elected leader, sending a full tablet report...
I20260808 02:10:26.195043 25862 ts_tablet_manager.cc:1428] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260808 02:10:26.196100 25865 ts_tablet_manager.cc:1428] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260808 02:10:26.198439 25482 tablet.cc:2369] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:10:26.316051 25490 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:26.316357 25490 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.317785 25490 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
I20260808 02:10:26.322314 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.317996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:33031 (local address 127.5.209.65:37897)
0808 02:10:26.318352 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.318368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.318469 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.318782 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:26.318787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:26.318810 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:26.318935 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.318941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.319732 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.319736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.320657 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.320667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.321707 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.321714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.321920 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.321966 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:10:26.322079 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:10:26.322159 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":109,"thread_start_us":108,"threads_started":1}
I20260808 02:10:26.322393 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.318306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:39539 (local address 127.5.209.67:46093)
0808 02:10:26.318437 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.318444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.318462 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:10:26.318688 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.318691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.318762 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.318847 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:26.319161 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.319640 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.320785 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.321432 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.321563 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.321624 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.322180 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.322265 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:10:26.322398 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.318320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37897 (local address 127.5.209.66:33031)
0808 02:10:26.318453 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.318458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.318472 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:10:26.318612 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.318617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.318686 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.318748 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:26.319073 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.319617 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.320818 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.321430 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.321566 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.321623 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.322127 (+ 504us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.322233 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:10:26.322782 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.318049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46093 (local address 127.5.209.65:39539)
0808 02:10:26.318396 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.318410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.318513 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.318890 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:26.318892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:26.318909 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:26.319032 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.319038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.319782 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.319784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.320652 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:26.320662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.321701 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.321707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:26.321990 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.322024 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:10:26.322531 (+ 507us) client_negotiation.cc:240] Negotiation successful
0808 02:10:26.322610 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":131,"thread_start_us":71,"threads_started":1}
I20260808 02:10:26.323287 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56cd2e6b38c44ed980051bfb7d351c0" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da85cc8a2911479c9d19d4a7e7026323" is_pre_election: true
I20260808 02:10:26.323442 25784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56cd2e6b38c44ed980051bfb7d351c0" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bf9c770f574b4fbd39f042bba8a563" is_pre_election: true
I20260808 02:10:26.323652 25599 raft_consensus.cc:2454] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 0.
I20260808 02:10:26.323756 25784 raft_consensus.cc:2454] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 0.
I20260808 02:10:26.324198 25337 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17bf9c770f574b4fbd39f042bba8a563, cbdb64a1a3664acdbe998e0c8986e49d, da85cc8a2911479c9d19d4a7e7026323; no voters:
I20260808 02:10:26.324472 25490 raft_consensus.cc:2781] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:10:26.324563 25490 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:10:26.324620 25490 raft_consensus.cc:3037] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:26.325707 25490 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.326035 25490 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 election: Requested vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
I20260808 02:10:26.326372 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56cd2e6b38c44ed980051bfb7d351c0" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da85cc8a2911479c9d19d4a7e7026323"
I20260808 02:10:26.326485 25784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e56cd2e6b38c44ed980051bfb7d351c0" candidate_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bf9c770f574b4fbd39f042bba8a563"
I20260808 02:10:26.326529 25599 raft_consensus.cc:3037] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:26.326637 25784 raft_consensus.cc:3037] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:26.327636 25599 raft_consensus.cc:2454] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 1.
I20260808 02:10:26.327775 25784 raft_consensus.cc:2454] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbdb64a1a3664acdbe998e0c8986e49d in term 1.
I20260808 02:10:26.327942 25339 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d, da85cc8a2911479c9d19d4a7e7026323; no voters:
I20260808 02:10:26.328131 25490 raft_consensus.cc:2781] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:26.328344 25490 raft_consensus.cc:686] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 LEADER]: Becoming Leader. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Running, Role: LEADER
I20260808 02:10:26.328616 25490 consensus_queue.cc:237] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.331902 25241 catalog_manager.cc:5495] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d reported cstate change: term changed from 0 to 1, leader changed from <none> to cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65). New cstate: current_term: 1 leader_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } health_report { overall_health: HEALTHY } } }
I20260808 02:10:26.370915 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.365388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.5.209.65:39839)
0808 02:10:26.366441 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:26.366453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:26.366472 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:10:26.366571 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:26.366577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:26.366664 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:26.366816 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:10:26.367140 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.367791 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.369050 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:26.369495 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:26.369633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:26.370023 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:10:26.370499 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:10:26.370595 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:26.370730 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 02:10:26.370781 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":802,"threads_started":1}
W20260808 02:10:26.375360 25849 tablet.cc:2369] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:10:26.376060 25784 raft_consensus.cc:1261] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Refusing update from remote peer cbdb64a1a3664acdbe998e0c8986e49d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:26.376130 25599 raft_consensus.cc:1261] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Refusing update from remote peer cbdb64a1a3664acdbe998e0c8986e49d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:26.376830 25873 consensus_queue.cc:1035] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:26.377034 25490 consensus_queue.cc:1035] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:26.394135 25241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:26.394315 25241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:26.396847 25241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Sending DeleteTablet for 3 replicas of tablet e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:26.397517 25576 tablet_service.cc:1514] Processing DeleteTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:26 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:55354
I20260808 02:10:26.397524 25759 tablet_service.cc:1514] Processing DeleteTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:26 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:38856
I20260808 02:10:26.398016 25889 tablet_replica.cc:330] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: stopping tablet replica
I20260808 02:10:26.398022 25893 tablet_replica.cc:330] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: stopping tablet replica
I20260808 02:10:26.398343 25893 raft_consensus.cc:2229] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:26.398360 25398 tablet_service.cc:1514] Processing DeleteTablet for tablet e56cd2e6b38c44ed980051bfb7d351c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:26 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48270
I20260808 02:10:26.398378 25889 raft_consensus.cc:2229] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:26.398660 25893 raft_consensus.cc:2258] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:26.398885 25889 raft_consensus.cc:2258] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:26.399274 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 25286
I20260808 02:10:26.404531 25889 ts_tablet_manager.cc:1905] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:26.408396 25889 ts_tablet_manager.cc:1918] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:26.408543 25889 log.cc:1196] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:26.409062 25889 ts_tablet_manager.cc:1939] T e56cd2e6b38c44ed980051bfb7d351c0 P da85cc8a2911479c9d19d4a7e7026323: Deleting consensus metadata
I20260808 02:10:26.410223 25893 ts_tablet_manager.cc:1905] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:26.410476 25224 catalog_manager.cc:4841] TS da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): tablet e56cd2e6b38c44ed980051bfb7d351c0 (table test-table [id=cff9cbc67f21488481ce7c1680ee747c]) successfully deleted
I20260808 02:10:26.412484 25893 ts_tablet_manager.cc:1918] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:26.412626 25893 log.cc:1196] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:26.413033 25893 ts_tablet_manager.cc:1939] T e56cd2e6b38c44ed980051bfb7d351c0 P 17bf9c770f574b4fbd39f042bba8a563: Deleting consensus metadata
I20260808 02:10:26.414489 25219 catalog_manager.cc:4841] TS 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): tablet e56cd2e6b38c44ed980051bfb7d351c0 (table test-table [id=cff9cbc67f21488481ce7c1680ee747c]) successfully deleted
W20260808 02:10:26.417557 25222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:39839 (error 108)
W20260808 02:10:26.418781 25488 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:10:26.418426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:39839 (local address 127.0.0.1:48294)
0808 02:10:26.418629 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.418698 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39839: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 02:10:26.418967 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 25487
W20260808 02:10:26.418975 25222 catalog_manager.cc:4584] TS cbdb64a1a3664acdbe998e0c8986e49d (127.5.209.65:39839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e56cd2e6b38c44ed980051bfb7d351c0: Network error: Client connection negotiation failed: client connection to 127.5.209.65:39839: connect: Connection reset by peer (error 104)
I20260808 02:10:26.434556 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 25668
I20260808 02:10:26.446717 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 25194
I20260808 02:10:26.460680 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--webserver_interface=127.5.209.126
--webserver_port=38469
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:26.581076 25897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:26.581453 25897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:26.581513 25897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:26.585430 25897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:26.585531 25897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:26.585551 25897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:26.585568 25897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:26.585583 25897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:26.590857 25897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=38469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:26.591878 25897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:26.592891 25897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:26.601533 25903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.601533 25906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.601553 25904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:26.602034 25897 server_base.cc:1034] running on GCE node
I20260808 02:10:26.602581 25897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:26.603744 25897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:26.604947 25897 hybrid_clock.cc:648] HybridClock initialized: now 1786155026604912 us; error 50 us; skew 500 ppm
Aug 08 02:10:26 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:26.607825 25897 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:26.608798 25897 webserver.cc:460] Webserver started at http://127.5.209.126:38469/ using document root <none> and password file <none>
I20260808 02:10:26.609443 25897 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:26.609539 25897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:26.613627 25897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:10:26.615823 25917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:26.617102 25897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:10:26.617249 25897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "451016ede2bc4dae96b4e689509441f3"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "2496a966f7552d674746fde94ff0fd0d"
server_key_iv: "e709602d585bee77c8e1713dd94695d9"
server_key_version: "encryptionkey@0"
I20260808 02:10:26.617887 25897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:26.642921 25897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:26.646425 25897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:26.646657 25897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:26.653892 25897 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34765
I20260808 02:10:26.654198 25984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34765 every 8 connection(s)
I20260808 02:10:26.655045 25897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:26.659168 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25897
I20260808 02:10:26.659684 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:39839
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=36139
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:26.659935 25985 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:10:26.663017 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap starting.
I20260808 02:10:26.681893 25985 log.cc:824] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:26.693637 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:10:26.694036 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3: Bootstrap complete.
I20260808 02:10:26.698402 25985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:26.698853 25985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Initialized, Role: FOLLOWER
I20260808 02:10:26.699434 25985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:26.699589 25985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:26.699684 25985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:26.699810 25985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:10:26.701870 25985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:26.702222 25985 leader_election.cc:302] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451016ede2bc4dae96b4e689509441f3; no voters:
I20260808 02:10:26.702584 25985 leader_election.cc:288] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:10:26.702685 25991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:10:26.702894 25991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [term 3 LEADER]: Becoming Leader. State: Replica: 451016ede2bc4dae96b4e689509441f3, State: Running, Role: LEADER
I20260808 02:10:26.703194 25991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } }
I20260808 02:10:26.703884 25985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:26.704627 25992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:26.704772 25992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:26.705452 25994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451016ede2bc4dae96b4e689509441f3. Latest consensus state: current_term: 3 leader_uuid: "451016ede2bc4dae96b4e689509441f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451016ede2bc4dae96b4e689509441f3" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34765 } } }
I20260808 02:10:26.705654 25994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451016ede2bc4dae96b4e689509441f3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:26.705925 26000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:26.710249 26000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:26.710815 26000 catalog_manager.cc:1259] Loaded cluster ID: 436fcb16e4e34f05b9652531b4ed7d31
I20260808 02:10:26.710896 26000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 02:10:26.711855 26000 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3d1ffc69ad google::LogMessage::Fail() at ??:0
@ 0x7f3d1ffc988a google::LogMessage::SendToLog() at ??:0
@ 0x7f3d1ffc64de google::LogMessage::Flush() at ??:0
@ 0x7f3d1ffc7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3d2a909fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3d2a90ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3d2a907579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3d2a94b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3d222183d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3d22218c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3d2221a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3d22209512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3d2384d6db start_thread at ??:0
@ 0x7f3d1eac461f clone at ??:0
W20260808 02:10:26.794700 25987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:26.794956 25987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:26.795004 25987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:26.799253 25987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:26.799312 25987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:26.799394 25987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:26.804764 25987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:39839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=36139
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:26.805962 25987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:26.807049 25987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:26.817166 26044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.817175 26040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:26.817304 26041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:26.818277 25987 server_base.cc:1034] running on GCE node
I20260808 02:10:26.818756 25987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:26.819478 25987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:26.820670 25987 hybrid_clock.cc:648] HybridClock initialized: now 1786155026820638 us; error 49 us; skew 500 ppm
Aug 08 02:10:26 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:26.823647 25987 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:26.824738 25987 webserver.cc:460] Webserver started at http://127.5.209.65:36139/ using document root <none> and password file <none>
I20260808 02:10:26.825408 25987 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:26.825479 25987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:26.829725 25987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:10:26.832090 26053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:26.833537 25987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:10:26.833665 25987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "cbdb64a1a3664acdbe998e0c8986e49d"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "d1372bcfc63094ebe2769b785e0b2449"
server_key_iv: "6d4aac41bdc6e9226756e621be212d53"
server_key_version: "encryptionkey@0"
I20260808 02:10:26.834249 25987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:26.845515 25987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:26.848884 25987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:26.849108 25987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:26.849812 25987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:26.851002 26062 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:10:26.856838 25987 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:10:26.856945 25987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 02:10:26.857035 25987 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:10:26.858968 25987 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:10:26.859043 25987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260808 02:10:26.859436 26062 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap starting.
I20260808 02:10:26.870700 25987 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:39839
I20260808 02:10:26.870884 26204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:39839 every 8 connection(s)
I20260808 02:10:26.871912 25987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:26.878398 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 25987
I20260808 02:10:26.878849 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:33031
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=41615
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:26.883782 26062 log.cc:824] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:26.894346 26062 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:10:26.894831 26062 tablet_bootstrap.cc:492] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Bootstrap complete.
I20260808 02:10:26.896533 26062 ts_tablet_manager.cc:1397] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Time spent bootstrapping tablet: real 0.037s user 0.029s sys 0.004s
I20260808 02:10:26.900134 26062 raft_consensus.cc:348] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.900723 26062 raft_consensus.cc:729] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdb64a1a3664acdbe998e0c8986e49d, State: Initialized, Role: FOLLOWER
I20260808 02:10:26.901360 26062 consensus_queue.cc:260] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:26.902518 26062 ts_tablet_manager.cc:1428] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260808 02:10:26.948010 26207 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:10:26.874073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:39611)
0808 02:10:26.874608 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.874662 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:26.875485 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:26.947006 (+ 71521us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
W20260808 02:10:26.948705 26205 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34765 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34765: Client connection negotiation failed: client connection to 127.5.209.126:34765: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:10:26.949556 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.949225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:50009)
0808 02:10:26.949393 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.949486 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 02:10:26.950255 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:26.950031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:52661)
0808 02:10:26.950146 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:26.950199 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:10:26.950466 26205 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:10:27.002761 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:27.003033 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:27.003077 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:27.007067 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:27.007118 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:27.007176 26208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:27.012318 26208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:33031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=41615
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:27.013451 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:27.014590 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:27.024463 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:27.024483 26222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:27.024470 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:27.024716 26208 server_base.cc:1034] running on GCE node
I20260808 02:10:27.025161 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:27.025835 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:27.027035 26208 hybrid_clock.cc:648] HybridClock initialized: now 1786155027027010 us; error 35 us; skew 500 ppm
Aug 08 02:10:27 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155027, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:27.029750 26208 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:27.030695 26208 webserver.cc:460] Webserver started at http://127.5.209.66:41615/ using document root <none> and password file <none>
I20260808 02:10:27.031283 26208 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:27.031383 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:27.035292 26208 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260808 02:10:27.037441 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.038800 26208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:10:27.038957 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "da85cc8a2911479c9d19d4a7e7026323"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "8fb148b2b93f7c206f66790c56f3a39a"
server_key_iv: "f8905917a792f659920cf13747eb18d7"
server_key_version: "encryptionkey@0"
I20260808 02:10:27.039513 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:27.073846 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:27.076874 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:27.077102 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:27.077767 26208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:27.078658 26208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:27.078732 26208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.078809 26208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:27.078851 26208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.090078 26208 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:33031
I20260808 02:10:27.090158 26382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:33031 every 8 connection(s)
I20260808 02:10:27.091442 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:27.096194 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.094285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:36077)
0808 02:10:27.094930 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.095440 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":156,"threads_started":1}
W20260808 02:10:27.096647 26383 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34765 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34765: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
I20260808 02:10:27.097325 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.097068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:41863)
0808 02:10:27.097210 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.097261 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 02:10:27.097445 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 26208
I20260808 02:10:27.097927 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:46093
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:34765
--webserver_port=34041
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:10:27.098093 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.097842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:45861)
0808 02:10:27.097952 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.098003 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:10:27.098278 26383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:10:27.217677 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:27.217906 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:27.217947 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:27.221892 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:27.221942 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:27.221997 26386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:27.227105 26386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:46093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=34041
--tserver_master_addrs=127.5.209.126:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:27.228047 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:27.229007 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:27.238461 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:27.238482 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:27.238477 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:27.238813 26386 server_base.cc:1034] running on GCE node
I20260808 02:10:27.239215 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:27.239825 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:27.240984 26386 hybrid_clock.cc:648] HybridClock initialized: now 1786155027240956 us; error 43 us; skew 500 ppm
Aug 08 02:10:27 dist-test-slave-bw88 krb5kdc[24457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155027, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:27.243824 26386 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:27.244735 26386 webserver.cc:460] Webserver started at http://127.5.209.67:34041/ using document root <none> and password file <none>
I20260808 02:10:27.245401 26386 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:27.245471 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:27.249224 26386 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:10:27.251263 26405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.252440 26386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:10:27.252594 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "17bf9c770f574b4fbd39f042bba8a563"
format_stamp: "Formatted at 2026-08-08 02:10:24 on dist-test-slave-bw88"
server_key: "4fe555cb6c92d9429626a157b3eef133"
server_key_iv: "ab9ae360499a8d4bb83f7bfeb31c1ef8"
server_key_version: "encryptionkey@0"
I20260808 02:10:27.253139 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:27.270347 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:27.273098 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:27.273311 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:27.273955 26386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:27.274819 26386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:27.274893 26386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.274966 26386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:27.275019 26386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:27.284687 26386 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:46093
I20260808 02:10:27.284720 26566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:46093 every 8 connection(s)
I20260808 02:10:27.285753 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:27.286094 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 26386
I20260808 02:10:27.290443 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.288734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:59625)
0808 02:10:27.289218 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.289724 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
W20260808 02:10:27.291020 26567 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34765 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34765: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
I20260808 02:10:27.291666 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.291431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:47527)
0808 02:10:27.291545 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.291614 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 02:10:27.292289 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.292098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:51043)
0808 02:10:27.292194 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.292237 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 02:10:27.292469 26567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:10:27.952183 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:27.951387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:59363)
0808 02:10:27.951857 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:27.951973 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260808 02:10:28.099766 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.099070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:51077)
0808 02:10:28.099493 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:28.099591 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
I20260808 02:10:28.294514 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.293589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:58007)
0808 02:10:28.294158 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:28.294291 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":248,"threads_started":1}
I20260808 02:10:28.425137 26574 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:28.425554 26574 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:28.427508 26574 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
Aug 08 02:10:28 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.66@KRBTEST.COM
Aug 08 02:10:28 dist-test-slave-bw88 krb5kdc[24457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.67@KRBTEST.COM
I20260808 02:10:28.459537 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.427839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:46093 (local address 127.5.209.65:53429)
0808 02:10:28.428535 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:28.428565 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:28.428732 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:28.432852 (+ 4120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:28.432862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:28.434324 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:28.436329 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:28.436358 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.440240 (+ 3882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.440251 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:28.443436 (+ 3185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:28.443461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.445424 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.445430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:28.446205 (+ 775us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:28.447506 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:28.447564 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:28.451648 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:28.455589 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:28.455609 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:28.455612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:28.456025 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:28.456648 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:28.456652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:28.456662 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:28.456840 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:28.458138 (+ 1298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:28.458142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:28.458341 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 02:10:28.458631 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 02:10:28.459196 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1002,"spinlock_wait_cycles":91392,"thread_start_us":149,"threads_started":1}
I20260808 02:10:28.459579 26577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.428285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:53429 (local address 127.5.209.67:46093)
0808 02:10:28.428986 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:28.429047 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:28.429128 (+ 81us) server_negotiation.cc:407] Connection header received
0808 02:10:28.430990 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:28.431020 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:28.432081 (+ 1061us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:28.432592 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:28.436638 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.439547 (+ 2909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.443803 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.445159 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.445508 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:28.452004 (+ 6496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:28.452019 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:28.452041 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:28.452130 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:28.455364 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:28.456191 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:28.456202 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:28.456209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:28.456434 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:28.457024 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:28.457028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:28.457030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:28.457816 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:28.458180 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:28.458838 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 02:10:28.459304 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":214,"threads_started":1}
I20260808 02:10:28.459695 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.427997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:42545 (local address 127.5.209.66:33031)
0808 02:10:28.428986 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:28.429047 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:28.429128 (+ 81us) server_negotiation.cc:407] Connection header received
0808 02:10:28.430990 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:28.431020 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:28.432081 (+ 1061us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:28.432559 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:28.436638 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.439548 (+ 2910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.443655 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.445159 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.445510 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:28.451596 (+ 6086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:28.451604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:28.451628 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:28.451678 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:28.454938 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:28.455920 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:28.455926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:28.455940 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:28.456128 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:28.456810 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:28.456814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:28.456815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:28.457674 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:28.457924 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:28.458876 (+ 952us) server_negotiation.cc:299] Negotiation successful
0808 02:10:28.459359 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":159,"threads_started":1}
I20260808 02:10:28.459759 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.427792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:33031 (local address 127.5.209.65:42545)
0808 02:10:28.428309 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:28.428338 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:28.428539 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:28.432852 (+ 4313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:28.432862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:28.434324 (+ 1462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:28.436329 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:28.436358 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.439809 (+ 3451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.439825 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:28.443436 (+ 3611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:28.443461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:28.445424 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:28.445430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:28.446209 (+ 779us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:28.447537 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:28.447572 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:28.451175 (+ 3603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:28.455178 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:28.455195 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:28.455232 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:28.455709 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:28.456313 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:28.456316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:28.456318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:28.456593 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:28.457891 (+ 1298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:28.457900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:28.458315 (+ 415us) client_negotiation.cc:769] Sending connection context
0808 02:10:28.458634 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 02:10:28.459350 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":1687,"spinlock_wait_cycles":35456}
W20260808 02:10:28.465420 26054 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
W20260808 02:10:28.466549 26057 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:28.466679 26057 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d; no voters: 17bf9c770f574b4fbd39f042bba8a563, da85cc8a2911479c9d19d4a7e7026323
I20260808 02:10:28.467037 26574 raft_consensus.cc:2726] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:10:28.953621 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:28.953272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:49415)
0808 02:10:28.953467 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:28.953527 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 02:10:29.101117 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:29.100713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:45499)
0808 02:10:29.100963 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:29.101018 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260808 02:10:29.296005 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:29.295577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:39363)
0808 02:10:29.295843 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:29.295899 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260808 02:10:29.954957 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:29.954474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:39335)
0808 02:10:29.954795 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:29.954853 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":132,"threads_started":1}
I20260808 02:10:30.102329 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:30.101920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:48701)
0808 02:10:30.102179 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:30.102232 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260808 02:10:30.297184 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:30.296728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:58449)
0808 02:10:30.297019 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:30.297078 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260808 02:10:30.605150 26583 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:30.605294 26583 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:30.606068 26583 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
W20260808 02:10:30.607038 26054 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
W20260808 02:10:30.607128 26057 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:30.607173 26054 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d; no voters: 17bf9c770f574b4fbd39f042bba8a563, da85cc8a2911479c9d19d4a7e7026323
I20260808 02:10:30.607312 26583 raft_consensus.cc:2726] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:10:30.956153 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:30.955692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:56631)
0808 02:10:30.955939 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:30.955991 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260808 02:10:31.103700 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:31.103260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:42121)
0808 02:10:31.103535 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:31.103602 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260808 02:10:31.298394 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:31.297973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:39953)
0808 02:10:31.298244 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:31.298299 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260808 02:10:31.957724 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:31.957179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:60513)
0808 02:10:31.957553 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:31.957623 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":77,"threads_started":1}
I20260808 02:10:32.105242 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:32.104699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:39967)
0808 02:10:32.105009 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:32.105105 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260808 02:10:32.299888 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:32.299298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:35993)
0808 02:10:32.299683 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:32.299753 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260808 02:10:32.958981 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:32.958541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:46089)
0808 02:10:32.958817 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:32.958875 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260808 02:10:33.106472 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:33.106054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:51467)
0808 02:10:33.106314 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:33.106373 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260808 02:10:33.301191 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:33.300724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:53387)
0808 02:10:33.301012 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:33.301073 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260808 02:10:33.662490 26595 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:33.662662 26595 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:33.663141 26595 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
W20260808 02:10:33.663965 26054 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
W20260808 02:10:33.664067 26057 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:33.664139 26057 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d; no voters: 17bf9c770f574b4fbd39f042bba8a563, da85cc8a2911479c9d19d4a7e7026323
I20260808 02:10:33.664286 26595 raft_consensus.cc:2726] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:10:33.960223 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:33.959763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:47511)
0808 02:10:33.960049 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:33.960121 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260808 02:10:34.107664 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:34.107204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:58921)
0808 02:10:34.107493 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:34.107547 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260808 02:10:34.302502 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:34.301980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:45233)
0808 02:10:34.302344 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:34.302399 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260808 02:10:34.961586 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:34.961065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:41489)
0808 02:10:34.961415 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:34.961481 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
I20260808 02:10:35.108899 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:35.108463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:41487)
0808 02:10:35.108740 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:35.108799 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260808 02:10:35.303800 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:35.303254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:49487)
0808 02:10:35.303578 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:35.303672 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260808 02:10:35.963028 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:35.962581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:37993)
0808 02:10:35.962872 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:35.962930 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
I20260808 02:10:36.110193 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:36.109712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:59529)
0808 02:10:36.110022 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:36.110082 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260808 02:10:36.305296 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:36.304736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:48483)
0808 02:10:36.305093 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:36.305168 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260808 02:10:36.964322 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:36.963897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:40457)
0808 02:10:36.964164 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:36.964222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260808 02:10:37.111471 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:37.111030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:45915)
0808 02:10:37.111309 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:37.111370 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260808 02:10:37.306552 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:37.306128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:47111)
0808 02:10:37.306396 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:37.306455 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260808 02:10:37.965608 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:37.965110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:45101)
0808 02:10:37.965428 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:37.965487 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260808 02:10:38.112962 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:38.112436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:41013)
0808 02:10:38.112737 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:38.112814 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260808 02:10:38.307943 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:38.307441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:46703)
0808 02:10:38.307763 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:38.307831 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260808 02:10:38.630846 26615 raft_consensus.cc:482] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:38.630992 26615 raft_consensus.cc:504] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da85cc8a2911479c9d19d4a7e7026323" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 33031 } } peers { permanent_uuid: "17bf9c770f574b4fbd39f042bba8a563" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 46093 } } peers { permanent_uuid: "cbdb64a1a3664acdbe998e0c8986e49d" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 39839 } }
I20260808 02:10:38.631505 26615 leader_election.cc:288] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031), 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093)
W20260808 02:10:38.632314 26054 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17bf9c770f574b4fbd39f042bba8a563 (127.5.209.67:46093): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
W20260808 02:10:38.632478 26057 leader_election.cc:341] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da85cc8a2911479c9d19d4a7e7026323 (127.5.209.66:33031): Not found: Tablet not found: e56cd2e6b38c44ed980051bfb7d351c0
I20260808 02:10:38.632579 26057 leader_election.cc:302] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cbdb64a1a3664acdbe998e0c8986e49d; no voters: 17bf9c770f574b4fbd39f042bba8a563, da85cc8a2911479c9d19d4a7e7026323
I20260808 02:10:38.632797 26615 raft_consensus.cc:2726] T e56cd2e6b38c44ed980051bfb7d351c0 P cbdb64a1a3664acdbe998e0c8986e49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:10:38.966833 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:38.966392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:40699)
0808 02:10:38.966672 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:38.966730 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260808 02:10:39.114192 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:39.113758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:41591)
0808 02:10:39.114039 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:39.114094 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260808 02:10:39.309142 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:39.308699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:42213)
0808 02:10:39.308988 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:39.309043 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260808 02:10:39.968122 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:39.967661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:43511)
0808 02:10:39.967949 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:39.968010 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260808 02:10:40.115367 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:40.114931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:58701)
0808 02:10:40.115211 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:40.115270 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260808 02:10:40.310315 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:40.309902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:37349)
0808 02:10:40.310167 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:40.310220 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260808 02:10:40.969560 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:40.969024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:52371)
0808 02:10:40.969396 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:40.969457 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260808 02:10:41.116585 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:41.116157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:55187)
0808 02:10:41.116436 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:41.116490 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260808 02:10:41.311410 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:41.311000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:37769)
0808 02:10:41.311255 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:41.311311 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260808 02:10:41.970954 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:41.970543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.65:44153)
0808 02:10:41.970800 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:41.970855 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260808 02:10:42.117915 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:42.117510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.66:60217)
0808 02:10:42.117759 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:42.117816 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260808 02:10:42.291653 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 25987
I20260808 02:10:42.313072 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:42.312623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34765 (local address 127.5.209.67:44543)
0808 02:10:42.312912 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:42.312974 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260808 02:10:42.319769 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 26208
I20260808 02:10:42.332031 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 26386
2026-08-08T02:10:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19614 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:10:42 dist-test-slave-bw88 krb5kdc[26636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:10:42 dist-test-slave-bw88 krb5kdc[26636](info): set up 2 sockets
Aug 08 02:10:42 dist-test-slave-bw88 krb5kdc[26636](info): commencing operation
krb5kdc: starting...
W20260808 02:10:43.543411 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:10:43 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:10:43.676448 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:43.795015 26653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:43.795320 26653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:43.795404 26653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:43.799484 26653 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:10:43.799556 26653 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:10:43.799590 26653 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:10:43.799624 26653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:10:43.799661 26653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:43.799718 26653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:43.799758 26653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:10:43.799794 26653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:10:43.804980 26653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:43.806124 26653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:43.807199 26653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:10:43.816390 26653 server_base.cc:1034] running on GCE node
W20260808 02:10:43.816341 26659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:43.816413 26658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:43.816342 26661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:43.817287 26653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:43.818436 26653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:43.819597 26653 hybrid_clock.cc:648] HybridClock initialized: now 1786155043819577 us; error 31 us; skew 500 ppm
Aug 08 02:10:43 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155043, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:43.822392 26653 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:10:43.823283 26653 webserver.cc:460] Webserver started at http://127.5.209.126:43789/ using document root <none> and password file <none>
I20260808 02:10:43.823810 26653 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:43.823868 26653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:43.824033 26653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:43.826097 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "9903e4b712db4f509b028d61ba8a33cd"
format_stamp: "Formatted at 2026-08-08 02:10:43 on dist-test-slave-bw88"
server_key: "d3cf35b7e4c6fcfc66e8e767558a8083"
server_key_iv: "56aad74ea78bb26b12aba5f7f1786065"
server_key_version: "encryptionkey@0"
I20260808 02:10:43.826618 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "9903e4b712db4f509b028d61ba8a33cd"
format_stamp: "Formatted at 2026-08-08 02:10:43 on dist-test-slave-bw88"
server_key: "d3cf35b7e4c6fcfc66e8e767558a8083"
server_key_iv: "56aad74ea78bb26b12aba5f7f1786065"
server_key_version: "encryptionkey@0"
I20260808 02:10:43.830262 26653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:10:43.832636 26672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:43.833972 26653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:10:43.834122 26653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9903e4b712db4f509b028d61ba8a33cd"
format_stamp: "Formatted at 2026-08-08 02:10:43 on dist-test-slave-bw88"
server_key: "d3cf35b7e4c6fcfc66e8e767558a8083"
server_key_iv: "56aad74ea78bb26b12aba5f7f1786065"
server_key_version: "encryptionkey@0"
I20260808 02:10:43.834237 26653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:43.860491 26653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:43.863543 26653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:43.863770 26653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:43.871605 26653 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34709
I20260808 02:10:43.871889 26732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34709 every 8 connection(s)
I20260808 02:10:43.872694 26653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:10:43.873651 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 26653
I20260808 02:10:43.873795 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:10:43.874161 5957 external_mini_cluster.cc:1428] Setting key f9e51f9dceecd6d64cc2cd4d7fa0aaa9
I20260808 02:10:43.876385 26733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:43.883414 26733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap starting.
Aug 08 02:10:43 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:43.886027 26733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:43.886809 26733 log.cc:824] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:43.888947 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:43.875628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.5.209.126:34709)
0808 02:10:43.876088 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:43.876119 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:43.876158 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:10:43.876894 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:43.876914 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:43.877273 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:43.877635 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:43.878563 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:43.879590 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:43.880734 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:43.881231 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:43.881418 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:43.884292 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:43.884297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:43.884311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:43.884340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:43.886048 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:43.886533 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:43.886538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:43.886543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:43.886660 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:43.886948 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:43.886954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:43.886956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:43.887381 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:43.887591 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:43.887903 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 02:10:43.888209 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":160,"threads_started":1}
I20260808 02:10:43.889083 26733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: No bootstrap required, opened a new log
I20260808 02:10:43.893237 26733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:10:43.893535 26733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:43.893606 26733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Initialized, Role: FOLLOWER
I20260808 02:10:43.894024 26733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:10:43.894173 26733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:10:43.894251 26733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:10:43.894348 26733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:43.895239 26733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:10:43.895577 26733 leader_election.cc:302] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9903e4b712db4f509b028d61ba8a33cd; no voters:
I20260808 02:10:43.895910 26733 leader_election.cc:288] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:10:43.896039 26739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:43.896318 26739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 LEADER]: Becoming Leader. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Running, Role: LEADER
I20260808 02:10:43.896679 26739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:10:43.897172 26733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:10:43.898519 26741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:10:43.898726 26741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:43.899190 26742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9903e4b712db4f509b028d61ba8a33cd. Latest consensus state: current_term: 1 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:10:43.899305 26742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:10:43.899247 26751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:10:43.902570 26751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:10:43.908264 26751 catalog_manager.cc:1347] Generated new cluster ID: 6f5e9845db184891aa5a0672c1877e7a
I20260808 02:10:43.908334 26751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:10:43.931996 26751 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:10:43.933287 26751 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:10:43.944554 26751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Generated new TSK 0
I20260808 02:10:43.945264 26751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:10:43.985021 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:44.102170 26768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:44.102418 26768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:44.102459 26768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:44.106328 26768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:44.106379 26768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:44.106433 26768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:10:44.111483 26768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:44.112427 26768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:44.113431 26768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:44.123159 26773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:44.123170 26776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:44.123148 26774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:44.123422 26768 server_base.cc:1034] running on GCE node
I20260808 02:10:44.123832 26768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:44.124475 26768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:44.125658 26768 hybrid_clock.cc:648] HybridClock initialized: now 1786155044125647 us; error 33 us; skew 500 ppm
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:44.128427 26768 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:10:44.129304 26768 webserver.cc:460] Webserver started at http://127.5.209.65:36733/ using document root <none> and password file <none>
I20260808 02:10:44.129930 26768 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:44.130018 26768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:44.130251 26768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:44.132267 26768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "a1b15bea67b74025a69e4bfa8516a333"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "53ac63d7a0dc3b3bda3cf1fb9425f094"
server_key_iv: "a7080aef1e971b1767b19a1ba6a9d38b"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.132833 26768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "a1b15bea67b74025a69e4bfa8516a333"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "53ac63d7a0dc3b3bda3cf1fb9425f094"
server_key_iv: "a7080aef1e971b1767b19a1ba6a9d38b"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.136531 26768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 02:10:44.139093 26788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.140347 26768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:10:44.140476 26768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "a1b15bea67b74025a69e4bfa8516a333"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "53ac63d7a0dc3b3bda3cf1fb9425f094"
server_key_iv: "a7080aef1e971b1767b19a1ba6a9d38b"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.140584 26768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:44.167929 26768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:44.171622 26768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:44.171839 26768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:44.172495 26768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:44.173442 26768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:44.173506 26768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.173578 26768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:44.173623 26768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.183081 26768 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:42335
I20260808 02:10:44.183115 26922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:42335 every 8 connection(s)
I20260808 02:10:44.184039 26768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:10:44.192095 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 26768
I20260808 02:10:44.192247 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:10:44.192572 5957 external_mini_cluster.cc:1428] Setting key 798649fd8af61111f016dbd1be0fdabe
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:44.198552 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.186170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:38541 (local address 127.5.209.126:34709)
0808 02:10:44.186321 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.186325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.187390 (+ 1065us) server_negotiation.cc:407] Connection header received
0808 02:10:44.188348 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.188352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.188410 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.188516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:44.189783 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.190327 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.191480 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.191719 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.191829 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.194559 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:44.194562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:44.194564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:44.194586 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:44.196051 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.196706 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.196712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.196716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.196819 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.197252 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.197260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.197262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.197521 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:44.197644 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.198222 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.198373 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:10:44.199453 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.186582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:38541)
0808 02:10:44.187245 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.187287 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.188141 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.188712 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.188719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.189085 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:44.189638 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.189653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.190457 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.190460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.191354 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.191363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.191874 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.191878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.192050 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.192686 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:44.192709 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:44.194389 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:44.196176 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.196185 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.196204 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.196603 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.196920 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.196922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.196924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.197141 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.197643 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:44.197650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:44.197919 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.198139 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.198563 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":110,"threads_started":1}
I20260808 02:10:44.200801 26924 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:10:44.201071 26924 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:44.201668 26924 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:10:44.203452 26690 ts_manager.cc:194] Registered new tserver with Master: a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:10:44.205024 26690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:38541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:10:44.223800 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:44.340030 26929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:44.340293 26929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:44.340335 26929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:44.344605 26929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:44.344676 26929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:44.344735 26929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:10:44.350461 26929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:44.351509 26929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:44.352543 26929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:44.362435 26935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:44.362716 26929 server_base.cc:1034] running on GCE node
W20260808 02:10:44.362452 26934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:44.362452 26937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:44.363257 26929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:44.363927 26929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:44.365089 26929 hybrid_clock.cc:648] HybridClock initialized: now 1786155044365061 us; error 51 us; skew 500 ppm
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:44.367934 26929 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:10:44.368821 26929 webserver.cc:460] Webserver started at http://127.5.209.66:39299/ using document root <none> and password file <none>
I20260808 02:10:44.369457 26929 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:44.369552 26929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:44.369800 26929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:44.371899 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "de8e2e863ff9444a972966b2cd8abf99"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "7cff00ef26f0c7e2a8b97ad28ac8c8d0"
server_key_iv: "ec8bbefd0bf3ff75f0e38b62b98f755a"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.372483 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "de8e2e863ff9444a972966b2cd8abf99"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "7cff00ef26f0c7e2a8b97ad28ac8c8d0"
server_key_iv: "ec8bbefd0bf3ff75f0e38b62b98f755a"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.376196 26929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 02:10:44.378749 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.379894 26929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:10:44.380030 26929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "de8e2e863ff9444a972966b2cd8abf99"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "7cff00ef26f0c7e2a8b97ad28ac8c8d0"
server_key_iv: "ec8bbefd0bf3ff75f0e38b62b98f755a"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.380174 26929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:44.442239 26929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:44.446039 26929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:44.446270 26929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:44.448721 26929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:44.449661 26929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:44.449712 26929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.449755 26929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:44.449780 26929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.459038 26929 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:37157
I20260808 02:10:44.459115 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:37157 every 8 connection(s)
I20260808 02:10:44.460000 26929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:10:44.461305 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 26929
I20260808 02:10:44.461467 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:10:44.461795 5957 external_mini_cluster.cc:1428] Setting key 56d52ac50cdaedc8829350f8a0e2e2fa
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:10:44.475581 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.461967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:32959 (local address 127.5.209.126:34709)
0808 02:10:44.462123 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.462128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.463445 (+ 1317us) server_negotiation.cc:407] Connection header received
0808 02:10:44.464800 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.464803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.464845 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.464920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:44.466492 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.467352 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.468535 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.468995 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.469139 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.471955 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:44.471962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:44.471968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:44.472003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:44.473671 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.474158 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.474161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.474162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.474237 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.474595 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.474598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.474600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.474789 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:44.474869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.475355 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.475456 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:10:44.476522 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.462466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:32959)
0808 02:10:44.463288 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.463350 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.464572 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.465134 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.465145 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.465750 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:44.466327 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.466342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.467479 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.467482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.468395 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.468404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.469167 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.469170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.469327 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.469980 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:44.469997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:44.471779 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:44.473787 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.473791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.473804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.474069 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.474316 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.474318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.474319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.474451 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.474891 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:44.474896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:44.475142 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.475309 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.475573 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":153,"threads_started":1}
I20260808 02:10:44.477852 27101 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:10:44.478130 27101 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:44.478708 27101 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:10:44.479791 26690 ts_manager.cc:194] Registered new tserver with Master: de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
I20260808 02:10:44.480392 26690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:32959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:10:44.493136 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:10:44.610282 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:10:44.610540 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:10:44.610613 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:10:44.614648 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:10:44.614705 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:10:44.614809 27109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:10:44.620087 27109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:10:44.621127 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:10:44.622219 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:10:44.632284 27114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:44.632275 27109 server_base.cc:1034] running on GCE node
W20260808 02:10:44.632258 27115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:10:44.632295 27117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:10:44.632804 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:10:44.633476 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:10:44.634656 27109 hybrid_clock.cc:648] HybridClock initialized: now 1786155044634642 us; error 28 us; skew 500 ppm
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:10:44.637452 27109 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:10:44.638320 27109 webserver.cc:460] Webserver started at http://127.5.209.67:32923/ using document root <none> and password file <none>
I20260808 02:10:44.638932 27109 fs_manager.cc:359] Metadata directory not provided
I20260808 02:10:44.638996 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:10:44.639228 27109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:10:44.641381 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "ae339a4b69674a64900fd01367bace1c"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "6ffba4ea95ff2f0aa17505878367ea73"
server_key_iv: "7b7a81f5463a23a7ea8fc6f83e595bae"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.641959 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "ae339a4b69674a64900fd01367bace1c"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "6ffba4ea95ff2f0aa17505878367ea73"
server_key_iv: "7b7a81f5463a23a7ea8fc6f83e595bae"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.645771 27109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:10:44.648193 27129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.649464 27109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:10:44.649580 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "ae339a4b69674a64900fd01367bace1c"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "6ffba4ea95ff2f0aa17505878367ea73"
server_key_iv: "7b7a81f5463a23a7ea8fc6f83e595bae"
server_key_version: "encryptionkey@0"
I20260808 02:10:44.649729 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:10:44.676736 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:10:44.679734 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:10:44.679942 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:10:44.680585 27109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:10:44.681499 27109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:10:44.681546 27109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.681622 27109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:10:44.681656 27109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:10:44.691100 27109 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:33931
I20260808 02:10:44.691134 27282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:33931 every 8 connection(s)
I20260808 02:10:44.692029 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:10:44.700618 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 27109
I20260808 02:10:44.700773 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:10:44.701105 5957 external_mini_cluster.cc:1428] Setting key 45d18ec0bfd505208b5f2fada94dc059
Aug 08 02:10:44 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155044, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:10:44.705928 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.693969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:57663 (local address 127.5.209.126:34709)
0808 02:10:44.694150 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.694155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.695202 (+ 1047us) server_negotiation.cc:407] Connection header received
0808 02:10:44.696088 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.696090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.696145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.696218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:44.697556 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.698087 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.699417 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.699631 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.699711 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.702546 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:44.702550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:44.702553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:44.702583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:44.704047 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.704524 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.704527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.704528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.704607 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.704900 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.704902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.704903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.705092 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:44.705169 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.705704 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.705806 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:10:44.706753 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.694373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:57663)
0808 02:10:44.695050 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.695096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.695902 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.696391 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.696398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.696805 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:10:44.697404 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.697423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.698214 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.698217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.699312 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.699322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.699766 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.699768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.699913 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.700561 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:10:44.700577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:10:44.702378 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:10:44.704164 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.704170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.704182 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.704438 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.704686 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:10:44.704688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:10:44.704689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:10:44.704821 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:10:44.705180 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:10:44.705184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:10:44.705487 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.705651 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.705905 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":110,"threads_started":1}
I20260808 02:10:44.707827 27283 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:10:44.708096 27283 heartbeater.cc:464] Registering TS with master...
I20260808 02:10:44.708616 27283 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:10:44.709677 26690 ts_manager.cc:194] Registered new tserver with Master: ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931)
I20260808 02:10:44.710256 26690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:57663
I20260808 02:10:44.714953 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:10:44.723951 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.716950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.5.209.126:34709)
0808 02:10:44.717064 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.717067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.717230 (+ 163us) server_negotiation.cc:407] Connection header received
0808 02:10:44.717319 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.717321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.717400 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.717466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:10:44.718152 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.718806 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.719598 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.719800 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.719904 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.721137 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:10:44.721140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:10:44.721152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:10:44.721175 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:10:44.722616 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.722966 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.722969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.722970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.723044 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:10:44.723260 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:10:44.723262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:10:44.723263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:10:44.723463 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:10:44.723568 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.723746 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.723836 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:10:44.726811 26690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:10:44.729939 26690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:10:44.745029 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.738467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:37157 (local address 127.0.0.1:33046)
0808 02:10:44.739166 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.739214 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.739348 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.740473 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.740477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.740496 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:44.740609 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.740617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.741805 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.741811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.743395 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.743405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.744456 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.744460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.744594 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.744641 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.744733 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.744851 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":125,"thread_start_us":101,"threads_started":1}
I20260808 02:10:44.745226 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.738597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33931 (local address 127.0.0.1:54160)
0808 02:10:44.739243 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.739257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.739373 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.740237 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.740243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.740287 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:44.740425 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.740432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.741954 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.741956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.743392 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.743403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.744818 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.744822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.744906 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.744928 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.744992 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.745058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":64,"thread_start_us":83,"threads_started":1}
I20260808 02:10:44.745620 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.738756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.5.209.67:33931)
0808 02:10:44.739683 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.739700 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.739724 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:10:44.739807 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.739814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.740028 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.740218 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:44.740565 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.741819 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.744063 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.744695 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.744815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.744969 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.745022 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.745456 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":168,"threads_started":1}
I20260808 02:10:44.746085 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.738623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.5.209.66:37157)
0808 02:10:44.739963 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.739974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.739995 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:10:44.740067 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.740075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.740284 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.740500 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:44.740716 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.741681 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.743518 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.744322 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.745616 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.745776 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.745830 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.745936 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1238,"thread_start_us":157,"threads_started":1}
I20260808 02:10:44.746151 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.739404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.0.0.1:53758)
0808 02:10:44.739928 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.739943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.740028 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.740818 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.740820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.740841 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:44.740935 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.740941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.743064 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.743067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.743931 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.743939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.745094 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.745098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.745210 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.745235 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.745303 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.745396 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":94,"threads_started":1}
I20260808 02:10:44.746308 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.739575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.5.209.65:42335)
0808 02:10:44.740356 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.740362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.740386 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:10:44.740467 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.740476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.740683 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.740831 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:44.741051 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.742163 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.744045 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.744875 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.745907 (+ 1032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.746042 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.746091 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.746175 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":85,"threads_started":1}
I20260808 02:10:44.748766 27196 tablet_service.cc:1467] Processing CreateTablet for tablet 271b5adc93134aad909e5392ed608c02 (DEFAULT_TABLE table=test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:44.748914 27019 tablet_service.cc:1467] Processing CreateTablet for tablet 271b5adc93134aad909e5392ed608c02 (DEFAULT_TABLE table=test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:44.750041 27019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b5adc93134aad909e5392ed608c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:44.750052 27196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b5adc93134aad909e5392ed608c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:44.749749 26844 tablet_service.cc:1467] Processing CreateTablet for tablet 271b5adc93134aad909e5392ed608c02 (DEFAULT_TABLE table=test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:10:44.750679 26844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271b5adc93134aad909e5392ed608c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:10:44.756500 27305 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Bootstrap starting.
I20260808 02:10:44.757937 27306 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap starting.
I20260808 02:10:44.758306 27307 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Bootstrap starting.
I20260808 02:10:44.759508 27305 tablet_bootstrap.cc:654] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:44.760349 27307 tablet_bootstrap.cc:654] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:44.760366 27306 tablet_bootstrap.cc:654] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Neither blocks nor log segments found. Creating new log.
I20260808 02:10:44.760936 27305 log.cc:824] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:44.761193 27306 log.cc:824] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:44.761505 27307 log.cc:824] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:10:44.763440 27305 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: No bootstrap required, opened a new log
I20260808 02:10:44.763489 27306 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: No bootstrap required, opened a new log
I20260808 02:10:44.763661 27305 ts_tablet_manager.cc:1397] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 02:10:44.763676 27306 ts_tablet_manager.cc:1397] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:10:44.763912 27307 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: No bootstrap required, opened a new log
I20260808 02:10:44.764086 27307 ts_tablet_manager.cc:1397] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260808 02:10:44.766769 27305 raft_consensus.cc:348] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.767179 27305 raft_consensus.cc:374] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:44.767295 27305 raft_consensus.cc:729] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e2e863ff9444a972966b2cd8abf99, State: Initialized, Role: FOLLOWER
I20260808 02:10:44.767778 27305 consensus_queue.cc:260] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.768114 27306 raft_consensus.cc:348] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.768064 27307 raft_consensus.cc:348] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.768335 27307 raft_consensus.cc:374] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:44.768399 27306 raft_consensus.cc:374] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:10:44.768433 27307 raft_consensus.cc:729] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae339a4b69674a64900fd01367bace1c, State: Initialized, Role: FOLLOWER
I20260808 02:10:44.768440 27101 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
I20260808 02:10:44.768492 27306 raft_consensus.cc:729] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b15bea67b74025a69e4bfa8516a333, State: Initialized, Role: FOLLOWER
I20260808 02:10:44.768814 27305 ts_tablet_manager.cc:1428] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260808 02:10:44.768961 27307 consensus_queue.cc:260] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.768961 27306 consensus_queue.cc:260] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.769668 26924 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
I20260808 02:10:44.769936 27306 ts_tablet_manager.cc:1428] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 02:10:44.769950 27307 ts_tablet_manager.cc:1428] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260808 02:10:44.770288 27283 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
I20260808 02:10:44.919052 27312 raft_consensus.cc:482] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:10:44.919310 27312 raft_consensus.cc:504] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.920750 27312 leader_election.cc:288] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
I20260808 02:10:44.924784 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.920995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33931 (local address 127.5.209.66:56525)
0808 02:10:44.921254 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.921272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.921422 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.921739 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.921742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.921761 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:44.921890 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.921895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.922631 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.922636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.923460 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.923470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.924320 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.924324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.924431 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.924460 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.924533 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.924622 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260808 02:10:44.924829 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.921133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:56525 (local address 127.5.209.67:33931)
0808 02:10:44.921294 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.921300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.921322 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:10:44.921559 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.921561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.921620 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.921721 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:44.922001 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.922508 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.923609 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.924190 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.924319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.924395 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.924578 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.924679 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:10:44.925007 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.921075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.5.209.66:53151)
0808 02:10:44.921553 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:44.921565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:10:44.921688 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:10:44.921943 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:10:44.921946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:10:44.921963 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:10:44.922098 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.922104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.922819 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.922822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.923752 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:10:44.923762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.924583 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.924586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:10:44.924693 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.924716 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:10:44.924790 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:10:44.924862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":113,"threads_started":1}
I20260808 02:10:44.925414 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.921202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:53151 (local address 127.5.209.65:42335)
0808 02:10:44.921361 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.921366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.921655 (+ 289us) server_negotiation.cc:407] Connection header received
0808 02:10:44.921802 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.921805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.921849 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.921931 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:10:44.922209 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.922705 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.923885 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.924434 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.924560 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.924608 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.925049 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.925277 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:10:44.925776 27220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b5adc93134aad909e5392ed608c02" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae339a4b69674a64900fd01367bace1c" is_pre_election: true
I20260808 02:10:44.926012 26866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b5adc93134aad909e5392ed608c02" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b15bea67b74025a69e4bfa8516a333" is_pre_election: true
I20260808 02:10:44.926121 27220 raft_consensus.cc:2454] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 0.
I20260808 02:10:44.926338 26866 raft_consensus.cc:2454] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 0.
I20260808 02:10:44.926733 26957 leader_election.cc:302] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99; no voters:
I20260808 02:10:44.927028 27312 raft_consensus.cc:2781] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:10:44.927122 27312 raft_consensus.cc:482] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:10:44.927168 27312 raft_consensus.cc:3037] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:44.928252 27312 raft_consensus.cc:504] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.928622 27312 leader_election.cc:288] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 election: Requested vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
I20260808 02:10:44.928963 27220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b5adc93134aad909e5392ed608c02" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae339a4b69674a64900fd01367bace1c"
I20260808 02:10:44.929000 26866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271b5adc93134aad909e5392ed608c02" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b15bea67b74025a69e4bfa8516a333"
I20260808 02:10:44.929097 27220 raft_consensus.cc:3037] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:44.929116 26866 raft_consensus.cc:3037] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:10:44.930500 26866 raft_consensus.cc:2454] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 1.
I20260808 02:10:44.930500 27220 raft_consensus.cc:2454] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 1.
I20260808 02:10:44.930851 26957 leader_election.cc:302] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99; no voters:
I20260808 02:10:44.931077 27312 raft_consensus.cc:2781] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:10:44.931342 27312 raft_consensus.cc:686] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 1 LEADER]: Becoming Leader. State: Replica: de8e2e863ff9444a972966b2cd8abf99, State: Running, Role: LEADER
I20260808 02:10:44.931733 27312 consensus_queue.cc:237] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:10:44.935263 26689 catalog_manager.cc:5495] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 reported cstate change: term changed from 0 to 1, leader changed from <none> to de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "de8e2e863ff9444a972966b2cd8abf99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } health_report { overall_health: UNKNOWN } } }
W20260808 02:10:44.935534 26925 tablet.cc:2369] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:10:44.942947 27284 tablet.cc:2369] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:10:44.961189 27102 tablet.cc:2369] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:10:44.979080 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:44.975386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.5.209.66:37157)
0808 02:10:44.975533 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 02:10:44.975538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:10:44.975553 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:10:44.975688 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:10:44.975690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:10:44.975749 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:10:44.975863 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:10:44.976105 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.976663 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.977588 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:10:44.977815 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:10:44.977922 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:10:44.978238 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:10:44.978679 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:10:44.978812 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:10:44.978853 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 02:10:44.978913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:10:44.985565 27220 raft_consensus.cc:1261] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Refusing update from remote peer de8e2e863ff9444a972966b2cd8abf99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:44.985565 26866 raft_consensus.cc:1261] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Refusing update from remote peer de8e2e863ff9444a972966b2cd8abf99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:10:44.986336 27312 consensus_queue.cc:1035] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:44.986558 27318 consensus_queue.cc:1035] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:10:44.994040 27322 mvcc.cc:205] Tried to move back new op lower bound from 7316091064252170240 to 7316091064043134976. Current Snapshot: MvccSnapshot[applied={T|T < 7316091064252170240}]
I20260808 02:10:44.995883 27324 mvcc.cc:205] Tried to move back new op lower bound from 7316091064252170240 to 7316091064043134976. Current Snapshot: MvccSnapshot[applied={T|T < 7316091064252170240}]
I20260808 02:10:44.997617 27323 mvcc.cc:205] Tried to move back new op lower bound from 7316091064252170240 to 7316091064043134976. Current Snapshot: MvccSnapshot[applied={T|T < 7316091064252170240 or (T in {7316091064252170240})}]
I20260808 02:10:45.003428 26688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:45.003612 26688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:10:45.006013 26688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Sending DeleteTablet for 3 replicas of tablet 271b5adc93134aad909e5392ed608c02
I20260808 02:10:45.006806 27019 tablet_service.cc:1514] Processing DeleteTablet for tablet 271b5adc93134aad909e5392ed608c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:33046
I20260808 02:10:45.006820 27196 tablet_service.cc:1514] Processing DeleteTablet for tablet 271b5adc93134aad909e5392ed608c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:54160
I20260808 02:10:45.006927 26844 tablet_service.cc:1514] Processing DeleteTablet for tablet 271b5adc93134aad909e5392ed608c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:53758
I20260808 02:10:45.007512 27332 tablet_replica.cc:330] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: stopping tablet replica
I20260808 02:10:45.007540 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 26768
I20260808 02:10:45.018216 27333 tablet_replica.cc:330] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: stopping tablet replica
I20260808 02:10:45.018790 27332 raft_consensus.cc:2229] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:10:45.019238 27333 raft_consensus.cc:2229] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:10:45.019624 27333 raft_consensus.cc:2258] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:45.020059 27332 raft_consensus.cc:2258] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:10:45.023372 27333 ts_tablet_manager.cc:1905] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:45.027215 27333 ts_tablet_manager.cc:1918] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:45.028302 27332 ts_tablet_manager.cc:1905] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:10:45.031342 27332 ts_tablet_manager.cc:1918] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:10:45.031888 27332 log.cc:1196] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/271b5adc93134aad909e5392ed608c02
I20260808 02:10:45.032250 27332 ts_tablet_manager.cc:1939] T 271b5adc93134aad909e5392ed608c02 P de8e2e863ff9444a972966b2cd8abf99: Deleting consensus metadata
I20260808 02:10:45.032374 27333 log.cc:1196] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/271b5adc93134aad909e5392ed608c02
I20260808 02:10:45.032822 27333 ts_tablet_manager.cc:1939] T 271b5adc93134aad909e5392ed608c02 P ae339a4b69674a64900fd01367bace1c: Deleting consensus metadata
I20260808 02:10:45.034320 26673 catalog_manager.cc:4841] TS de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): tablet 271b5adc93134aad909e5392ed608c02 (table test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]) successfully deleted
I20260808 02:10:45.034827 26677 catalog_manager.cc:4841] TS ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): tablet 271b5adc93134aad909e5392ed608c02 (table test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]) successfully deleted
I20260808 02:10:45.043594 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 26929
W20260808 02:10:45.043578 26676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:42335 (error 108)
I20260808 02:10:45.049757 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:10:45.049470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.0.0.1:53764)
0808 02:10:45.049608 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 02:10:45.049674 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:42335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:10:45.049930 26676 catalog_manager.cc:4584] TS a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 271b5adc93134aad909e5392ed608c02: Network error: Client connection negotiation failed: client connection to 127.5.209.65:42335: connect: Connection refused (error 111)
I20260808 02:10:45.055749 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 27109
I20260808 02:10:45.066015 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 26653
I20260808 02:11:05.075842 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--webserver_interface=127.5.209.126
--webserver_port=43789
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:05.201720 27337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:05.202033 27337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:05.202107 27337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:05.206301 27337 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:11:05.206370 27337 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:11:05.206398 27337 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:11:05.206434 27337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:05.206475 27337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:05.206569 27337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:05.206629 27337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:05.206669 27337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:05.211890 27337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=43789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:05.212980 27337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:05.214169 27337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:11:05.228865 27337 server_base.cc:1034] running on GCE node
W20260808 02:11:05.228878 27347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.228888 27348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.228879 27351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:05.229822 27337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:05.231045 27337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:05.232241 27337 hybrid_clock.cc:648] HybridClock initialized: now 1786155065232224 us; error 29 us; skew 500 ppm
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:05.240367 27337 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:11:05.241426 27337 webserver.cc:460] Webserver started at http://127.5.209.126:43789/ using document root <none> and password file <none>
I20260808 02:11:05.242053 27337 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:05.242170 27337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:05.246173 27337 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:11:05.248355 27362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.249644 27337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:11:05.249799 27337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9903e4b712db4f509b028d61ba8a33cd"
format_stamp: "Formatted at 2026-08-08 02:10:43 on dist-test-slave-bw88"
server_key: "d3cf35b7e4c6fcfc66e8e767558a8083"
server_key_iv: "56aad74ea78bb26b12aba5f7f1786065"
server_key_version: "encryptionkey@0"
I20260808 02:11:05.250591 27337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:05.265914 27337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:05.269304 27337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:05.269591 27337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:05.277251 27337 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34709
I20260808 02:11:05.277571 27433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34709 every 8 connection(s)
I20260808 02:11:05.278520 27337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:05.283211 27434 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:11:05.284821 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 27337
I20260808 02:11:05.285027 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap starting.
I20260808 02:11:05.285445 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:42335
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=36733
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:11:05.304445 27434 log.cc:824] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:05.314077 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:11:05.314531 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap complete.
I20260808 02:11:05.319160 27434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:05.319555 27434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Initialized, Role: FOLLOWER
I20260808 02:11:05.320122 27434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:05.320271 27434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:05.320369 27434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:05.320504 27434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:11:05.322628 27434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:05.323000 27434 leader_election.cc:302] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9903e4b712db4f509b028d61ba8a33cd; no voters:
I20260808 02:11:05.323398 27434 leader_election.cc:288] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:11:05.323486 27441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:11:05.323743 27441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 LEADER]: Becoming Leader. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Running, Role: LEADER
I20260808 02:11:05.324123 27441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:05.324748 27434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:05.325655 27444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:11:05.325794 27444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:05.326666 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9903e4b712db4f509b028d61ba8a33cd. Latest consensus state: current_term: 2 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:11:05.326833 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:05.326869 27454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:05.331167 27454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:05.331717 27454 catalog_manager.cc:1259] Loaded cluster ID: 6f5e9845db184891aa5a0672c1877e7a
I20260808 02:11:05.331801 27454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:05.333724 27454 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:05.334337 27454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Loaded TSK: 0
I20260808 02:11:05.350472 27454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Generated new TSK 1
I20260808 02:11:05.350757 27454 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:11:05.426810 27439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:05.427106 27439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:05.427204 27439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:05.431248 27439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:05.431341 27439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:05.431509 27439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:05.437029 27439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:42335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=36733
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:05.438176 27439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:05.439258 27439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:05.450918 27471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.450918 27476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.450968 27473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:05.451480 27439 server_base.cc:1034] running on GCE node
I20260808 02:11:05.451874 27439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:05.452608 27439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:05.453787 27439 hybrid_clock.cc:648] HybridClock initialized: now 1786155065453753 us; error 50 us; skew 500 ppm
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:05.456744 27439 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:11:05.457772 27439 webserver.cc:460] Webserver started at http://127.5.209.65:36733/ using document root <none> and password file <none>
I20260808 02:11:05.458348 27439 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:05.458425 27439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:05.462500 27439 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:05.464811 27486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.466189 27439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:05.466286 27439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "a1b15bea67b74025a69e4bfa8516a333"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "53ac63d7a0dc3b3bda3cf1fb9425f094"
server_key_iv: "a7080aef1e971b1767b19a1ba6a9d38b"
server_key_version: "encryptionkey@0"
I20260808 02:11:05.466889 27439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:05.491667 27439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:05.495602 27439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:05.495808 27439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:05.496469 27439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:05.497676 27494 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:11:05.502420 27439 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:11:05.502491 27439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260808 02:11:05.502544 27439 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:11:05.504276 27439 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:11:05.504338 27439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 02:11:05.504619 27494 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap starting.
I20260808 02:11:05.516211 27439 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:42335
I20260808 02:11:05.516346 27628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:42335 every 8 connection(s)
I20260808 02:11:05.517584 27439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:05.524230 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 27439
I20260808 02:11:05.524847 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:37157
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=39299
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:11:05.528546 27494 log.cc:824] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:05.538502 27494 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:11:05.538580 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.519988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:48195 (local address 127.5.209.126:34709)
0808 02:11:05.520396 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.520421 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.520824 (+ 403us) server_negotiation.cc:407] Connection header received
0808 02:11:05.522113 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.522126 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.522474 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.522838 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.524165 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.525997 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.527806 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.528330 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.528533 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.531870 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.531877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.531908 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.531946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.534403 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.535149 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.535157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.535164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.535328 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.535799 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.535803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.535805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.536290 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.536450 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.537205 (+ 755us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.537583 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":142,"threads_started":1}
I20260808 02:11:05.538692 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.519999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:48195)
0808 02:11:05.520628 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.520674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.521647 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.522905 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.522914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.523380 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:05.523918 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.523940 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.526204 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.526212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.527646 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.527658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.528493 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.528498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.528746 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.529481 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:05.529502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:05.531621 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:05.534561 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.534570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.534589 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.535002 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.535458 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.535461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.535463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.535671 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.536439 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:05.536449 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:05.536801 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.537052 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.537483 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":144,"threads_started":1}
I20260808 02:11:05.539075 27494 tablet_bootstrap.cc:492] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap complete.
I20260808 02:11:05.540732 27494 ts_tablet_manager.cc:1397] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.010s
I20260808 02:11:05.541656 27629 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:11:05.541903 27629 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:05.542448 27629 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:11:05.543552 27494 raft_consensus.cc:348] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:05.544123 27494 raft_consensus.cc:729] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b15bea67b74025a69e4bfa8516a333, State: Initialized, Role: FOLLOWER
I20260808 02:11:05.544660 27494 consensus_queue.cc:260] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:05.545228 27387 ts_manager.cc:194] Registered new tserver with Master: a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
I20260808 02:11:05.545764 27494 ts_tablet_manager.cc:1428] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 02:11:05.549321 27387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:48195
I20260808 02:11:05.552103 27629 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.65@KRBTEST.COM
I20260808 02:11:05.560217 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.548718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.5.209.65:42335)
0808 02:11:05.549522 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.549534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.549559 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:11:05.549624 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.549628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.549784 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.549954 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.550737 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.551939 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.553130 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.553569 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.553707 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.556429 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.556439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.556450 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.556473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.558149 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.558707 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.558713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.558714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.558796 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.559114 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.559120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.559122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.559366 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.559526 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.559888 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.560060 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":691,"thread_start_us":101,"threads_started":1}
I20260808 02:11:05.560209 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.548666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.0.0.1:35230)
0808 02:11:05.549079 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.549125 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.549258 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.549950 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.549957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.550442 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:05.550598 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.550608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.552090 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.552094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.552995 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.553005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.553730 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.553734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.553865 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.554541 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:05.554554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:05.556253 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:05.558275 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.558287 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.558294 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.558556 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.558902 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.558907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.558909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.559009 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.559498 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:05.559505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:05.559640 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.559796 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.560010 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
I20260808 02:11:05.561539 27548 tablet_service.cc:1514] Processing DeleteTablet for tablet 271b5adc93134aad909e5392ed608c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:10:45 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:35230
I20260808 02:11:05.562459 27642 tablet_replica.cc:330] stopping tablet replica
I20260808 02:11:05.562788 27642 raft_consensus.cc:2229] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:05.563058 27642 raft_consensus.cc:2258] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:05.564227 27642 ts_tablet_manager.cc:1905] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:05.568981 27642 ts_tablet_manager.cc:1918] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:05.569092 27642 log.cc:1196] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal/wals/271b5adc93134aad909e5392ed608c02
I20260808 02:11:05.569415 27642 ts_tablet_manager.cc:1939] T 271b5adc93134aad909e5392ed608c02 P a1b15bea67b74025a69e4bfa8516a333: Deleting consensus metadata
I20260808 02:11:05.571590 27365 catalog_manager.cc:4841] TS a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335): tablet 271b5adc93134aad909e5392ed608c02 (table test-table [id=13a89a5eb7a7400e94b56995cd8b5e7a]) successfully deleted
W20260808 02:11:05.656479 27636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:05.656739 27636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:05.656782 27636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:05.660845 27636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:05.660905 27636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:05.660964 27636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:05.666131 27636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:37157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=39299
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:05.667115 27636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:05.668100 27636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:05.678041 27648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.678035 27647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.678040 27650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:05.678306 27636 server_base.cc:1034] running on GCE node
I20260808 02:11:05.678855 27636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:05.679601 27636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:05.680807 27636 hybrid_clock.cc:648] HybridClock initialized: now 1786155065680777 us; error 50 us; skew 500 ppm
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:05.683835 27636 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:11:05.684703 27636 webserver.cc:460] Webserver started at http://127.5.209.66:39299/ using document root <none> and password file <none>
I20260808 02:11:05.685284 27636 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:05.685441 27636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:05.689599 27636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:11:05.691649 27659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.692830 27636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:05.692981 27636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "de8e2e863ff9444a972966b2cd8abf99"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "7cff00ef26f0c7e2a8b97ad28ac8c8d0"
server_key_iv: "ec8bbefd0bf3ff75f0e38b62b98f755a"
server_key_version: "encryptionkey@0"
I20260808 02:11:05.693564 27636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:05.718418 27636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:05.721769 27636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:05.721978 27636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:05.722586 27636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:05.723428 27636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:05.723498 27636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.723572 27636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:05.723630 27636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.733376 27636 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:37157
I20260808 02:11:05.733443 27803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:37157 every 8 connection(s)
I20260808 02:11:05.734457 27636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:05.744635 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 27636
I20260808 02:11:05.745252 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:33931
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=32923
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:11:05.748137 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.736391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:60087 (local address 127.5.209.126:34709)
0808 02:11:05.736546 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.736550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.737686 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 02:11:05.738524 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.738527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.738584 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.738696 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.739869 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.740480 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.741700 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.741946 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.742049 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.744534 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.744537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.744539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.744563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.746037 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.746554 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.746558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.746560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.746653 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.746997 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.747002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.747004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.747187 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.747299 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.747820 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.747959 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:11:05.748906 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.736789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:60087)
0808 02:11:05.737524 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.737566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.738332 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.738868 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.738874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.739235 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:05.739705 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.739718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.740602 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.740605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.741581 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.741591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.742086 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.742089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.742233 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.742836 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:05.742852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:05.744354 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:05.746166 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.746172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.746187 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.746443 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.746752 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.746755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.746757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.746896 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.747300 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:05.747307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:05.747552 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.747741 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.748060 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":123,"threads_started":1}
I20260808 02:11:05.750170 27804 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:11:05.750427 27804 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:05.750926 27804 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:11:05.752143 27387 ts_manager.cc:194] Registered new tserver with Master: de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
I20260808 02:11:05.752718 27387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:60087
W20260808 02:11:05.872123 27808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:05.872417 27808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:05.872490 27808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:05.877058 27808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:05.877156 27808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:05.877247 27808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:05.883297 27808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:33931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=32923
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:05.884439 27808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:05.885586 27808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:11:05.895857 27808 server_base.cc:1034] running on GCE node
W20260808 02:11:05.895833 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.895823 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:05.895815 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:05.896522 27808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:05.897184 27808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:05.898350 27808 hybrid_clock.cc:648] HybridClock initialized: now 1786155065898321 us; error 52 us; skew 500 ppm
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:05.901161 27808 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:11:05.902117 27808 webserver.cc:460] Webserver started at http://127.5.209.67:32923/ using document root <none> and password file <none>
I20260808 02:11:05.902638 27808 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:05.902709 27808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:05.906602 27808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 02:11:05.908778 27826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.910163 27808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:05.910311 27808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "ae339a4b69674a64900fd01367bace1c"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "6ffba4ea95ff2f0aa17505878367ea73"
server_key_iv: "7b7a81f5463a23a7ea8fc6f83e595bae"
server_key_version: "encryptionkey@0"
I20260808 02:11:05.910873 27808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:05.929949 27808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:05.933004 27808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:05.933228 27808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:05.933898 27808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:05.934844 27808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:05.934916 27808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.934984 27808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:05.935029 27808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:05.945530 27808 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:33931
I20260808 02:11:05.945613 27972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:33931 every 8 connection(s)
I20260808 02:11:05.946657 27808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:05.955076 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 27808
Aug 08 02:11:05 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:05.961097 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.948582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:43657 (local address 127.5.209.126:34709)
0808 02:11:05.948737 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.948741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.949866 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 02:11:05.950748 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.950751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.950807 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.950888 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.952081 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.952651 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.953924 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.954159 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.954278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.957292 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.957296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.957299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.957326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.958851 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.959443 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.959446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.959447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.959535 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.959952 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.959955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.959957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.960148 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.960255 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.960786 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.960909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:11:05.961992 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.948941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:43657)
0808 02:11:05.949700 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.949742 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.950523 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.951060 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.951066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.951432 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:05.951933 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.951947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.952764 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.952767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.953790 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.953808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.954324 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.954329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.954512 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.955302 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:05.955319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:05.957121 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:05.958981 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.958987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.959009 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.959330 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.959636 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:05.959642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:05.959645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:05.959839 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:05.960272 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:05.960280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:05.960541 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.960741 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.961074 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":116,"threads_started":1}
I20260808 02:11:05.963374 27975 heartbeater.cc:347] Connected to a master server at 127.5.209.126:34709
I20260808 02:11:05.963653 27975 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:05.964298 27975 heartbeater.cc:511] Master 127.5.209.126:34709 requested a full tablet report, sending...
I20260808 02:11:05.964601 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.955870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.5.209.126:34709)
0808 02:11:05.956148 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.956152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.956372 (+ 220us) server_negotiation.cc:407] Connection header received
0808 02:11:05.956459 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.956465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.956531 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.956632 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.957412 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.958201 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.959231 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.959595 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.959705 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.960982 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.960986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.960989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.961015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.962721 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.963165 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.963168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.963170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.963265 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.963658 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.963661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.963662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.963880 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.964044 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.964287 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.964417 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
I20260808 02:11:05.965547 27387 ts_manager.cc:194] Registered new tserver with Master: ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931)
I20260808 02:11:05.966100 27387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:43657
I20260808 02:11:05.966435 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:05.970757 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.967086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.5.209.126:34709)
0808 02:11:05.967186 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.967190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.967487 (+ 297us) server_negotiation.cc:407] Connection header received
0808 02:11:05.967646 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.967649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.967750 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.967859 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:05.968106 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.968594 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.969539 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.969774 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.969890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.970215 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:05.970339 (+ 124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 02:11:05.970616 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42324: token expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 02:11:05.970815 27980 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42324: token expired
W20260808 02:11:05.971254 27289 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.209.126:34709)
I20260808 02:11:05.979192 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.971659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.5.209.126:34709)
0808 02:11:05.971777 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.971783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.971818 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:11:05.971945 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.971948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.971991 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.972082 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:05.972731 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.973217 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.974214 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.974422 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.974529 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.975918 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:05.975922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:05.975925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:05.975959 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:05.977522 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.977962 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.977968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.977974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.978050 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:05.978328 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:05.978331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:05.978333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:05.978509 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:05.978647 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.978902 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.979033 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:11:05.981721 27387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:05.985013 27387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:05.993454 27548 tablet_service.cc:1467] Processing CreateTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 (DEFAULT_TABLE table=test-table [id=eda01bdeadcd40c6a7c3094eade9125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:05.994182 27548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4ec2b50ab9420ca25070e9edc42b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:05.997292 27494 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap starting.
I20260808 02:11:05.997825 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.992168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33931 (local address 127.0.0.1:36386)
0808 02:11:05.992313 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.992331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.992427 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.993416 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.993421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.993445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:05.993537 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.993542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.994974 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.994981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.996095 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.996107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.997268 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.997271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.997431 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.997488 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.997541 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.997649 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:11:05.998236 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.992566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:37157 (local address 127.0.0.1:52752)
0808 02:11:05.992868 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:05.992882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:05.992985 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:05.993968 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:05.993971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:05.993991 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:05.994104 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.994111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.995607 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.995610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.996617 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:05.996628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.997787 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.997790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:05.997907 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.997934 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:11:05.998016 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:11:05.998101 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":134,"threads_started":1}
I20260808 02:11:05.998347 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.993063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.5.209.66:37157)
0808 02:11:05.993472 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.993488 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.993514 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:11:05.993596 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.993601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.993794 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.994258 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:05.994434 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.995474 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.996793 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.997654 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.997773 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.997976 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.998064 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.998159 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260808 02:11:05.998373 27494 tablet_bootstrap.cc:654] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:05.998529 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:05.992398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.5.209.67:33931)
0808 02:11:05.992797 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:05.992811 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:05.992838 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:11:05.992929 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:05.992935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:05.993201 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:05.993670 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:05.993881 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.994857 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.996242 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:05.997117 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:05.998096 (+ 979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:05.998249 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:05.998308 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 02:11:05.998414 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":162,"threads_started":1}
I20260808 02:11:06.000231 27494 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: No bootstrap required, opened a new log
I20260808 02:11:06.000344 27494 ts_tablet_manager.cc:1397] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:06.000831 27494 raft_consensus.cc:348] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.000969 27494 raft_consensus.cc:374] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:06.001017 27494 raft_consensus.cc:729] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b15bea67b74025a69e4bfa8516a333, State: Initialized, Role: FOLLOWER
I20260808 02:11:06.001240 27494 consensus_queue.cc:260] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.001562 27494 ts_tablet_manager.cc:1428] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 02:11:06.001807 27886 tablet_service.cc:1467] Processing CreateTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 (DEFAULT_TABLE table=test-table [id=eda01bdeadcd40c6a7c3094eade9125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:06.002236 27720 tablet_service.cc:1467] Processing CreateTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 (DEFAULT_TABLE table=test-table [id=eda01bdeadcd40c6a7c3094eade9125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:06.003100 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4ec2b50ab9420ca25070e9edc42b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:06.003191 27720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf4ec2b50ab9420ca25070e9edc42b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:06.010588 27992 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Bootstrap starting.
I20260808 02:11:06.011423 27991 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Bootstrap starting.
I20260808 02:11:06.013685 27992 tablet_bootstrap.cc:654] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:06.013706 27991 tablet_bootstrap.cc:654] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:06.014735 27992 log.cc:824] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:06.014964 27991 log.cc:824] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:06.017527 27992 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: No bootstrap required, opened a new log
I20260808 02:11:06.017642 27991 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: No bootstrap required, opened a new log
I20260808 02:11:06.017745 27992 ts_tablet_manager.cc:1397] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 02:11:06.017863 27991 ts_tablet_manager.cc:1397] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
W20260808 02:11:06.019053 27632 tablet.cc:2369] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:06.020853 27992 raft_consensus.cc:348] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.021131 27992 raft_consensus.cc:374] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:06.020967 27991 raft_consensus.cc:348] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.021224 27992 raft_consensus.cc:729] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e2e863ff9444a972966b2cd8abf99, State: Initialized, Role: FOLLOWER
I20260808 02:11:06.021248 27991 raft_consensus.cc:374] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:06.021315 27991 raft_consensus.cc:729] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae339a4b69674a64900fd01367bace1c, State: Initialized, Role: FOLLOWER
I20260808 02:11:06.021737 27992 consensus_queue.cc:260] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.021821 27991 consensus_queue.cc:260] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.022534 27804 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
I20260808 02:11:06.022786 27992 ts_tablet_manager.cc:1428] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 02:11:06.022912 27991 ts_tablet_manager.cc:1428] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260808 02:11:06.023223 27975 heartbeater.cc:503] Master 127.5.209.126:34709 was elected leader, sending a full tablet report...
W20260808 02:11:06.197808 27978 tablet.cc:2369] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:11:06.235812 27805 tablet.cc:2369] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:06.343892 27995 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:06.344178 27995 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.345552 27995 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
I20260808 02:11:06.349568 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.345954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:60555 (local address 127.5.209.67:33931)
0808 02:11:06.346128 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:06.346136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:06.346321 (+ 185us) server_negotiation.cc:407] Connection header received
0808 02:11:06.346522 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:06.346524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:06.346570 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:06.346667 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:06.346963 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.347472 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.348559 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.348963 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.349096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:06.349154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:06.349373 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 02:11:06.349444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:11:06.349582 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.345775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33931 (local address 127.5.209.66:60555)
0808 02:11:06.346222 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:06.346241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:06.346377 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:06.346683 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:06.346686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:06.346703 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:06.346838 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:06.346844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.347607 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.347611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:06.348424 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:06.348433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.349096 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.349099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:06.349215 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:06.349241 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:11:06.349333 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:11:06.349425 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":138,"threads_started":1}
I20260808 02:11:06.350054 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.346106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:39487 (local address 127.5.209.65:42335)
0808 02:11:06.346437 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:06.346442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:06.346459 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:11:06.346637 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:06.346640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:06.346710 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:06.346844 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:06.347160 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.347746 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.348909 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.349478 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.349623 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:06.349795 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:06.349838 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 02:11:06.349910 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260808 02:11:06.350057 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.345972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.5.209.66:39487)
0808 02:11:06.346332 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:06.346347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:06.346470 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:06.346856 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:06.346859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:06.346878 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:06.347021 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:06.347028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.347880 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.347885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:06.348762 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:06.348771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.349616 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.349620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:06.349722 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:06.349746 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:11:06.349817 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:11:06.349890 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260808 02:11:06.350409 27912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4ec2b50ab9420ca25070e9edc42b96" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae339a4b69674a64900fd01367bace1c" is_pre_election: true
I20260808 02:11:06.350756 27912 raft_consensus.cc:2454] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 0.
I20260808 02:11:06.350767 27575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4ec2b50ab9420ca25070e9edc42b96" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b15bea67b74025a69e4bfa8516a333" is_pre_election: true
I20260808 02:11:06.351084 27575 raft_consensus.cc:2454] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 0.
I20260808 02:11:06.351308 27663 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99; no voters:
I20260808 02:11:06.351641 27995 raft_consensus.cc:2781] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:06.351735 27995 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:06.351780 27995 raft_consensus.cc:3037] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:06.352869 27995 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.353240 27995 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 election: Requested vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335)
I20260808 02:11:06.353664 27912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4ec2b50ab9420ca25070e9edc42b96" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae339a4b69674a64900fd01367bace1c"
I20260808 02:11:06.353719 27575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf4ec2b50ab9420ca25070e9edc42b96" candidate_uuid: "de8e2e863ff9444a972966b2cd8abf99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b15bea67b74025a69e4bfa8516a333"
I20260808 02:11:06.353817 27912 raft_consensus.cc:3037] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:06.353859 27575 raft_consensus.cc:3037] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:06.354995 27912 raft_consensus.cc:2454] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 1.
I20260808 02:11:06.355072 27575 raft_consensus.cc:2454] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e2e863ff9444a972966b2cd8abf99 in term 1.
I20260808 02:11:06.355443 27663 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99; no voters:
I20260808 02:11:06.355675 27995 raft_consensus.cc:2781] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:06.355919 27995 raft_consensus.cc:686] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 1 LEADER]: Becoming Leader. State: Replica: de8e2e863ff9444a972966b2cd8abf99, State: Running, Role: LEADER
I20260808 02:11:06.356287 27995 consensus_queue.cc:237] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:06.359846 27384 catalog_manager.cc:5495] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 reported cstate change: term changed from 0 to 1, leader changed from <none> to de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "de8e2e863ff9444a972966b2cd8abf99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } health_report { overall_health: UNKNOWN } } }
I20260808 02:11:06.455415 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.451628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.5.209.66:37157)
0808 02:11:06.451795 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:06.451798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:06.451829 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:11:06.452018 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:06.452021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:06.452084 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:06.452182 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:06.452459 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.452961 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.453921 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:06.454148 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:06.454231 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:06.454525 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:06.455017 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:06.455147 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:06.455242 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 02:11:06.455296 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:11:06.462188 27575 raft_consensus.cc:1261] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Refusing update from remote peer de8e2e863ff9444a972966b2cd8abf99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:06.462193 27912 raft_consensus.cc:1261] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Refusing update from remote peer de8e2e863ff9444a972966b2cd8abf99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:06.462997 27995 consensus_queue.cc:1035] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:06.463199 28001 consensus_queue.cc:1035] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:06.479434 27384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:06.479637 27384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:06.482151 27384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Sending DeleteTablet for 3 replicas of tablet cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:06.482887 27886 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:06 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:36386
I20260808 02:11:06.482880 27720 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:06 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:52752
I20260808 02:11:06.483233 27548 tablet_service.cc:1514] Processing DeleteTablet for tablet cf4ec2b50ab9420ca25070e9edc42b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:06 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:35230
I20260808 02:11:06.483491 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 27439
I20260808 02:11:06.483650 28016 tablet_replica.cc:330] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: stopping tablet replica
I20260808 02:11:06.483804 28016 raft_consensus.cc:2229] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:06.483948 28016 raft_consensus.cc:2258] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:06.487094 28013 tablet_replica.cc:330] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: stopping tablet replica
I20260808 02:11:06.487439 28013 raft_consensus.cc:2229] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:06.487882 28013 raft_consensus.cc:2258] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:06.488233 28015 tablet_replica.cc:330] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: stopping tablet replica
I20260808 02:11:06.488607 28015 raft_consensus.cc:2229] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:06.488970 28015 raft_consensus.cc:2258] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:06.490483 28013 ts_tablet_manager.cc:1905] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:06.491623 28015 ts_tablet_manager.cc:1905] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:06.494527 28015 ts_tablet_manager.cc:1918] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:06.494556 28013 ts_tablet_manager.cc:1918] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:06.494670 28015 log.cc:1196] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:06.494804 28013 log.cc:1196] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:06.495007 28015 ts_tablet_manager.cc:1939] T cf4ec2b50ab9420ca25070e9edc42b96 P ae339a4b69674a64900fd01367bace1c: Deleting consensus metadata
I20260808 02:11:06.495286 28013 ts_tablet_manager.cc:1939] T cf4ec2b50ab9420ca25070e9edc42b96 P de8e2e863ff9444a972966b2cd8abf99: Deleting consensus metadata
I20260808 02:11:06.496505 27366 catalog_manager.cc:4841] TS ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): tablet cf4ec2b50ab9420ca25070e9edc42b96 (table test-table [id=eda01bdeadcd40c6a7c3094eade9125b]) successfully deleted
I20260808 02:11:06.496989 27363 catalog_manager.cc:4841] TS de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): tablet cf4ec2b50ab9420ca25070e9edc42b96 (table test-table [id=eda01bdeadcd40c6a7c3094eade9125b]) successfully deleted
W20260808 02:11:06.502074 27365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:42335 (error 108)
I20260808 02:11:06.502317 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 27636
I20260808 02:11:06.504905 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:06.504356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:42335 (local address 127.0.0.1:35242)
0808 02:11:06.504674 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:06.504767 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:42335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":59,"threads_started":1}
W20260808 02:11:06.505116 27365 catalog_manager.cc:4584] TS a1b15bea67b74025a69e4bfa8516a333 (127.5.209.65:42335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf4ec2b50ab9420ca25070e9edc42b96: Network error: Client connection negotiation failed: client connection to 127.5.209.65:42335: connect: Connection refused (error 111)
I20260808 02:11:06.513484 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 27808
I20260808 02:11:06.530233 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 27337
I20260808 02:11:26.542586 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--webserver_interface=127.5.209.126
--webserver_port=43789
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:26.660238 28019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:26.660529 28019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:26.660602 28019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:26.665971 28019 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:11:26.666051 28019 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:11:26.666077 28019 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:11:26.666111 28019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:26.666148 28019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:26.666188 28019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:26.666224 28019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:26.666260 28019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:26.671586 28019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=43789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:26.672681 28019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:26.674160 28019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:11:26.683452 28019 server_base.cc:1034] running on GCE node
W20260808 02:11:26.683400 28030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:26.683394 28027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:26.683400 28026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:26.684239 28019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:26.685290 28019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:26.686523 28019 hybrid_clock.cc:648] HybridClock initialized: now 1786155086686510 us; error 27 us; skew 500 ppm
Aug 08 02:11:26 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155086, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:26.694157 28019 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:11:26.695128 28019 webserver.cc:460] Webserver started at http://127.5.209.126:43789/ using document root <none> and password file <none>
I20260808 02:11:26.695714 28019 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:26.695808 28019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:26.699781 28019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:11:26.702136 28039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:26.703581 28019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:26.703743 28019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9903e4b712db4f509b028d61ba8a33cd"
format_stamp: "Formatted at 2026-08-08 02:10:43 on dist-test-slave-bw88"
server_key: "d3cf35b7e4c6fcfc66e8e767558a8083"
server_key_iv: "56aad74ea78bb26b12aba5f7f1786065"
server_key_version: "encryptionkey@0"
I20260808 02:11:26.704336 28019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:26.721237 28019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:26.725006 28019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:26.725219 28019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:26.732434 28019 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34709
I20260808 02:11:26.732781 28105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34709 every 8 connection(s)
I20260808 02:11:26.733502 28019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:26.737975 28106 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:11:26.739625 28106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap starting.
I20260808 02:11:26.740747 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28019
I20260808 02:11:26.741436 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:42335
--local_ip_for_outbound_sockets=127.5.209.65
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=36733
--webserver_interface=127.5.209.65
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:11:26.758337 28106 log.cc:824] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:26.770550 28106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:11:26.770982 28106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Bootstrap complete.
I20260808 02:11:26.775355 28106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:26.775749 28106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Initialized, Role: FOLLOWER
I20260808 02:11:26.776325 28106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:26.776450 28106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:26.776538 28106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:26.776664 28106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:11:26.778645 28106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:26.778991 28106 leader_election.cc:302] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9903e4b712db4f509b028d61ba8a33cd; no voters:
I20260808 02:11:26.779340 28106 leader_election.cc:288] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:11:26.779433 28111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:11:26.779671 28111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [term 3 LEADER]: Becoming Leader. State: Replica: 9903e4b712db4f509b028d61ba8a33cd, State: Running, Role: LEADER
I20260808 02:11:26.780045 28111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } }
I20260808 02:11:26.780624 28106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:26.781126 28113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9903e4b712db4f509b028d61ba8a33cd. Latest consensus state: current_term: 3 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:11:26.781262 28113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:26.781566 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9903e4b712db4f509b028d61ba8a33cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9903e4b712db4f509b028d61ba8a33cd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34709 } } }
I20260808 02:11:26.781678 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:26.781821 28118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:26.787210 28118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:26.787953 28118 catalog_manager.cc:1259] Loaded cluster ID: 6f5e9845db184891aa5a0672c1877e7a
I20260808 02:11:26.788020 28118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:26.790593 28118 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:26.791360 28118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Loaded TSK: 0
I20260808 02:11:26.791447 28118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9903e4b712db4f509b028d61ba8a33cd: Loaded TSK: 1
F20260808 02:11:26.792246 28118 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f57eedce9ad google::LogMessage::Fail() at ??:0
@ 0x7f57eedd188a google::LogMessage::SendToLog() at ??:0
@ 0x7f57eedce4de google::LogMessage::Flush() at ??:0
@ 0x7f57eedcfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f57f1a9218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f57f1a8af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f57f97377bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f57f971199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f57f9712178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f57f9753df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f57f977282b std::function<>::operator()() at ??:0
@ 0x7f57f9711b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f57f9712f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f57f970f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f57f9753640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f57f10203d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f57f1020c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f57f10223f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f57f1011512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f57f26556db start_thread at ??:0
@ 0x7f57ed8cc61f clone at ??:0
W20260808 02:11:26.901134 28109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:26.901549 28109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:26.901623 28109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:26.907954 28109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:26.908037 28109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:26.908131 28109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:26.916363 28109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:42335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=36733
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:26.917837 28109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:26.919225 28109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:26.930727 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:26.930771 28109 server_base.cc:1034] running on GCE node
W20260808 02:11:26.930735 28163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:26.930980 28167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:26.931430 28109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:26.932228 28109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:26.933425 28109 hybrid_clock.cc:648] HybridClock initialized: now 1786155086933389 us; error 52 us; skew 500 ppm
Aug 08 02:11:26 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155086, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:26.937043 28109 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:11:26.938035 28109 webserver.cc:460] Webserver started at http://127.5.209.65:36733/ using document root <none> and password file <none>
I20260808 02:11:26.938620 28109 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:26.938683 28109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:26.942834 28109 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:11:26.945209 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:26.946586 28109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:26.946724 28109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "a1b15bea67b74025a69e4bfa8516a333"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "53ac63d7a0dc3b3bda3cf1fb9425f094"
server_key_iv: "a7080aef1e971b1767b19a1ba6a9d38b"
server_key_version: "encryptionkey@0"
I20260808 02:11:26.947284 28109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:26.971359 28109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:26.974762 28109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:26.974977 28109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:26.975605 28109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:26.976851 28182 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 02:11:26.982910 28109 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 02:11:26.983006 28109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 02:11:26.983109 28109 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 02:11:26.984881 28109 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 02:11:26.984956 28109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 02:11:26.985306 28182 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap starting.
I20260808 02:11:26.996498 28109 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:42335
I20260808 02:11:26.997604 28321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:42335 every 8 connection(s)
I20260808 02:11:26.997709 28109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:27.000576 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28109
I20260808 02:11:27.001053 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:37157
--local_ip_for_outbound_sockets=127.5.209.66
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=39299
--webserver_interface=127.5.209.66
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:11:27.010264 28182 log.cc:824] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:27.022266 28182 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:11:27.022781 28182 tablet_bootstrap.cc:492] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Bootstrap complete.
I20260808 02:11:27.024506 28182 ts_tablet_manager.cc:1397] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.018s
I20260808 02:11:27.028090 28182 raft_consensus.cc:348] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:27.028698 28182 raft_consensus.cc:729] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b15bea67b74025a69e4bfa8516a333, State: Initialized, Role: FOLLOWER
I20260808 02:11:27.029363 28182 consensus_queue.cc:260] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:27.030539 28182 ts_tablet_manager.cc:1428] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260808 02:11:27.070123 28325 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:11:26.999820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:49667)
0808 02:11:27.001746 (+ 1926us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.001818 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:27.003323 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:27.068977 (+ 65654us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":155,"threads_started":1}
W20260808 02:11:27.070794 28323 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34709 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34709: Client connection negotiation failed: client connection to 127.5.209.126:34709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:11:27.071655 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.071350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:36699)
0808 02:11:27.071516 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.071590 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 02:11:27.072377 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.072147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:33595)
0808 02:11:27.072268 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.072320 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:11:27.072582 28323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:11:27.163568 28326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:27.163863 28326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:27.163954 28326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:27.168202 28326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:27.168282 28326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:27.168371 28326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:27.173574 28326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:37157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=39299
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:27.174681 28326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:27.175750 28326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:27.186841 28341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:27.186864 28334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:27.187083 28326 server_base.cc:1034] running on GCE node
W20260808 02:11:27.186827 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:27.187577 28326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:27.188323 28326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:27.189518 28326 hybrid_clock.cc:648] HybridClock initialized: now 1786155087189503 us; error 30 us; skew 500 ppm
Aug 08 02:11:27 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:27.192332 28326 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:11:27.193295 28326 webserver.cc:460] Webserver started at http://127.5.209.66:39299/ using document root <none> and password file <none>
I20260808 02:11:27.193943 28326 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:27.194046 28326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:27.198310 28326 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:11:27.200796 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.202314 28326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:11:27.202420 28326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "de8e2e863ff9444a972966b2cd8abf99"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "7cff00ef26f0c7e2a8b97ad28ac8c8d0"
server_key_iv: "ec8bbefd0bf3ff75f0e38b62b98f755a"
server_key_version: "encryptionkey@0"
I20260808 02:11:27.202981 28326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:27.230798 28326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:27.234062 28326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:27.234223 28326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:27.234786 28326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:27.235610 28326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:27.235658 28326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.235700 28326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:27.235716 28326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.244786 28326 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:37157
I20260808 02:11:27.244812 28497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:37157 every 8 connection(s)
I20260808 02:11:27.245893 28326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:27.249471 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28326
I20260808 02:11:27.249747 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.248107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:39165)
0808 02:11:27.248795 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.248898 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":109,"threads_started":1}
I20260808 02:11:27.250036 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:33931
--local_ip_for_outbound_sockets=127.5.209.67
--tserver_master_addrs=127.5.209.126:34709
--webserver_port=32923
--webserver_interface=127.5.209.67
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:27.250339 28498 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34709 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34709: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
I20260808 02:11:27.251025 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.250767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:58121)
0808 02:11:27.250923 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.250971 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 02:11:27.251860 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.251604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:46331)
0808 02:11:27.251748 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.251803 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 02:11:27.252135 28498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:11:27.366961 28504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:27.367211 28504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:27.367254 28504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:27.371217 28504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:27.371268 28504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:27.371323 28504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:27.376410 28504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:33931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=32923
--tserver_master_addrs=127.5.209.126:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:27.377447 28504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:27.378477 28504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:27.388240 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:27.388252 28514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:27.388289 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:27.388551 28504 server_base.cc:1034] running on GCE node
I20260808 02:11:27.388928 28504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:27.389639 28504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:27.390801 28504 hybrid_clock.cc:648] HybridClock initialized: now 1786155087390788 us; error 29 us; skew 500 ppm
Aug 08 02:11:27 dist-test-slave-bw88 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155087, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:27.393826 28504 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:11:27.394681 28504 webserver.cc:460] Webserver started at http://127.5.209.67:32923/ using document root <none> and password file <none>
I20260808 02:11:27.395231 28504 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:27.395295 28504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:27.399353 28504 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:11:27.401803 28523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.403119 28504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:11:27.403223 28504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "ae339a4b69674a64900fd01367bace1c"
format_stamp: "Formatted at 2026-08-08 02:10:44 on dist-test-slave-bw88"
server_key: "6ffba4ea95ff2f0aa17505878367ea73"
server_key_iv: "7b7a81f5463a23a7ea8fc6f83e595bae"
server_key_version: "encryptionkey@0"
I20260808 02:11:27.403772 28504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:27.425230 28504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:27.428767 28504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:27.428952 28504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:27.429687 28504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:27.430564 28504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:27.430619 28504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.430665 28504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:27.430682 28504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:27.440896 28504 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:33931
I20260808 02:11:27.440964 28677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:33931 every 8 connection(s)
I20260808 02:11:27.441993 28504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:27.446197 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.444350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:52483)
0808 02:11:27.445262 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.445443 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":253,"threads_started":1}
W20260808 02:11:27.446659 28678 heartbeater.cc:644] Failed to heartbeat to 127.5.209.126:34709 (0 consecutive failures): Network error: Failed to ping master at 127.5.209.126:34709: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
I20260808 02:11:27.447309 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.447066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:50813)
0808 02:11:27.447195 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.447254 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:11:27.447989 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:27.447736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:51169)
0808 02:11:27.447876 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:27.447936 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 02:11:27.448065 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28504
W20260808 02:11:27.448215 28678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:11:28.073911 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.073329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:50875)
0808 02:11:28.073693 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:28.073754 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260808 02:11:28.253089 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.252647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:48619)
0808 02:11:28.252925 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:28.252983 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260808 02:11:28.449196 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.448760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:45281)
0808 02:11:28.449038 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:28.449095 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260808 02:11:28.889238 28686 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:28.889566 28686 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:28.890782 28686 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
Aug 08 02:11:28 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155086, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.67@KRBTEST.COM
Aug 08 02:11:28 dist-test-slave-bw88 krb5kdc[26636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155086, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.66@KRBTEST.COM
I20260808 02:11:28.905404 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.891016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33931 (local address 127.5.209.65:33185)
0808 02:11:28.891374 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:28.891392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:28.891528 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:28.893311 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:28.893316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:28.893901 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:28.894370 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:28.894386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.895672 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.895675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:28.896588 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:28.896598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.897313 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.897317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:28.897579 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:28.898234 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:28.898251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:28.899820 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:28.902338 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:28.902345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:28.902360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:28.902766 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:28.903267 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:28.903277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:28.903454 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:28.904314 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:28.904319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:28.904571 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 02:11:28.904784 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 02:11:28.905235 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":311,"spinlock_wait_cycles":6784,"thread_start_us":84,"threads_started":1}
I20260808 02:11:28.905509 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.891190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:33185 (local address 127.5.209.67:33931)
0808 02:11:28.891726 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:28.891747 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:28.891780 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:11:28.892634 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:28.892642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:28.892948 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:28.893262 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:28.894559 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.895517 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.896716 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.897181 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.897382 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:28.900075 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:28.900082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:28.900100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:28.900141 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:28.902181 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:28.902937 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:28.902958 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:28.902967 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:28.903150 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903561 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:28.903565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:28.903566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:28.904186 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:28.904339 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:28.904952 (+ 613us) server_negotiation.cc:299] Negotiation successful
0808 02:11:28.905268 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":60,"threads_started":1}
I20260808 02:11:28.905570 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.891266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:53509 (local address 127.5.209.66:37157)
0808 02:11:28.891658 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:28.891675 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:28.891722 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:11:28.892599 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:28.892606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:28.892948 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:28.893262 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:28.894551 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.895519 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.896714 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.897175 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.897378 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:28.900409 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:28.900414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:28.900426 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:28.900520 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:28.902906 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903466 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:28.903471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:28.903476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:28.903657 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903942 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:28.903946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:28.903948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:28.904479 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:28.904589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:28.904943 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 02:11:28.905268 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260808 02:11:28.905404 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:28.891047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:37157 (local address 127.5.209.65:53509)
0808 02:11:28.891419 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:28.891435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:28.891544 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:28.893310 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:28.893316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:28.893901 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:28.894370 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:28.894385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.895667 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.895671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:28.896585 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:28.896595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:28.897313 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:28.897317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:28.897580 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:28.898290 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:28.898303 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:28.900196 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:28.903026 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:28.903033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:28.903313 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:28.903768 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:28.903770 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:28.903772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:28.903843 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:28.904588 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:28.904590 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:28.904675 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 02:11:28.904785 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:11:28.905174 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":127,"spinlock_wait_cycles":3456,"thread_start_us":85,"threads_started":1}
W20260808 02:11:28.908775 28175 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
W20260808 02:11:28.909036 28179 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:28.909169 28179 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b15bea67b74025a69e4bfa8516a333; no voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99
I20260808 02:11:28.909570 28686 raft_consensus.cc:2726] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:11:29.075057 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:29.074749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:50457)
0808 02:11:29.074901 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:29.074968 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 02:11:29.254401 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:29.253969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:51105)
0808 02:11:29.254232 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:29.254288 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260808 02:11:29.450451 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:29.449950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:48657)
0808 02:11:29.450223 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:29.450352 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260808 02:11:30.076327 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:30.075909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:53739)
0808 02:11:30.076174 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:30.076230 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260808 02:11:30.255612 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:30.255175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:38287)
0808 02:11:30.255453 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:30.255519 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260808 02:11:30.451737 28696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:30.451305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:34579)
0808 02:11:30.451584 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:30.451643 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260808 02:11:30.631065 28698 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:30.631210 28698 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:30.631762 28698 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
W20260808 02:11:30.632745 28179 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
W20260808 02:11:30.632858 28175 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:30.632925 28175 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b15bea67b74025a69e4bfa8516a333; no voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99
I20260808 02:11:30.633073 28698 raft_consensus.cc:2726] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:11:31.077724 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:31.077175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:51481)
0808 02:11:31.077554 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:31.077619 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":80,"threads_started":1}
I20260808 02:11:31.257053 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:31.256553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:52779)
0808 02:11:31.256887 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:31.256955 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260808 02:11:31.453104 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:31.452624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:52749)
0808 02:11:31.452932 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:31.453007 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260808 02:11:32.079320 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:32.078813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:40681)
0808 02:11:32.079150 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:32.079218 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":80,"threads_started":1}
I20260808 02:11:32.258476 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:32.258004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:47289)
0808 02:11:32.258310 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:32.258371 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260808 02:11:32.454478 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:32.454033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:51841)
0808 02:11:32.454321 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:32.454380 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260808 02:11:33.080660 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:33.080217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:48407)
0808 02:11:33.080496 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:33.080559 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260808 02:11:33.198269 28709 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:33.198453 28709 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:33.198879 28709 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
W20260808 02:11:33.199851 28175 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
W20260808 02:11:33.199971 28179 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:33.200037 28179 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b15bea67b74025a69e4bfa8516a333; no voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99
I20260808 02:11:33.200179 28709 raft_consensus.cc:2726] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:11:33.259824 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:33.259372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:33115)
0808 02:11:33.259664 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:33.259725 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260808 02:11:33.455736 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:33.455313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:33937)
0808 02:11:33.455581 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:33.455641 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260808 02:11:34.082187 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:34.081620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:37519)
0808 02:11:34.081993 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:34.082070 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260808 02:11:34.261108 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:34.260621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:35061)
0808 02:11:34.260941 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:34.260999 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
I20260808 02:11:34.457032 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:34.456570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:53883)
0808 02:11:34.456868 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:34.456925 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260808 02:11:35.083406 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:35.082991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:49479)
0808 02:11:35.083251 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:35.083308 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260808 02:11:35.262421 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:35.261970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:54051)
0808 02:11:35.262245 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:35.262302 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260808 02:11:35.458359 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:35.457912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:34797)
0808 02:11:35.458197 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:35.458264 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260808 02:11:36.084849 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:36.084318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:45015)
0808 02:11:36.084612 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:36.084676 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260808 02:11:36.263793 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:36.263266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:47113)
0808 02:11:36.263629 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:36.263694 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":87,"threads_started":1}
I20260808 02:11:36.459625 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:36.459136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:44835)
0808 02:11:36.459448 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:36.459516 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260808 02:11:36.502708 28726 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:36.502851 28726 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:36.503382 28726 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
W20260808 02:11:36.504201 28179 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
W20260808 02:11:36.504312 28175 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:36.504405 28179 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b15bea67b74025a69e4bfa8516a333; no voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99
I20260808 02:11:36.504593 28726 raft_consensus.cc:2726] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:11:37.086670 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:37.085961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:47597)
0808 02:11:37.086355 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:37.086444 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260808 02:11:37.265074 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:37.264613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:58219)
0808 02:11:37.264898 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:37.264976 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
I20260808 02:11:37.460877 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:37.460451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:57891)
0808 02:11:37.460726 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:37.460783 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260808 02:11:38.087970 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:38.087502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:34235)
0808 02:11:38.087794 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:38.087858 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260808 02:11:38.266331 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:38.265905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:42359)
0808 02:11:38.266173 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:38.266232 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260808 02:11:38.462342 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:38.461899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:45819)
0808 02:11:38.462183 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:38.462242 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260808 02:11:39.089435 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:39.088794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:58075)
0808 02:11:39.089187 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:39.089283 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260808 02:11:39.267627 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:39.267168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:41667)
0808 02:11:39.267443 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:39.267504 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260808 02:11:39.463680 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:39.463205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:43757)
0808 02:11:39.463518 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:39.463579 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260808 02:11:40.090718 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:40.090256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:33111)
0808 02:11:40.090554 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:40.090617 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260808 02:11:40.268853 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:40.268393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:44503)
0808 02:11:40.268686 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:40.268746 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
I20260808 02:11:40.464951 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:40.464493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:33489)
0808 02:11:40.464794 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:40.464852 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
I20260808 02:11:41.092054 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:41.091513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:40197)
0808 02:11:41.091836 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:41.091912 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260808 02:11:41.270159 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:41.269734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:51599)
0808 02:11:41.269993 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:41.270058 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260808 02:11:41.466200 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:41.465775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:56523)
0808 02:11:41.466029 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:41.466099 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260808 02:11:42.069645 28746 raft_consensus.cc:482] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:42.069802 28746 raft_consensus.cc:504] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae339a4b69674a64900fd01367bace1c" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33931 } } peers { permanent_uuid: "de8e2e863ff9444a972966b2cd8abf99" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 37157 } } peers { permanent_uuid: "a1b15bea67b74025a69e4bfa8516a333" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 42335 } }
I20260808 02:11:42.070363 28746 leader_election.cc:288] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931), de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157)
W20260808 02:11:42.071275 28179 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae339a4b69674a64900fd01367bace1c (127.5.209.67:33931): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
W20260808 02:11:42.071448 28175 leader_election.cc:341] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer de8e2e863ff9444a972966b2cd8abf99 (127.5.209.66:37157): Not found: Tablet not found: cf4ec2b50ab9420ca25070e9edc42b96
I20260808 02:11:42.071563 28175 leader_election.cc:302] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b15bea67b74025a69e4bfa8516a333; no voters: ae339a4b69674a64900fd01367bace1c, de8e2e863ff9444a972966b2cd8abf99
I20260808 02:11:42.071762 28746 raft_consensus.cc:2726] T cf4ec2b50ab9420ca25070e9edc42b96 P a1b15bea67b74025a69e4bfa8516a333 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 02:11:42.093580 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:42.093116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.65:36863)
0808 02:11:42.093423 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:42.093484 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":125,"threads_started":1}
I20260808 02:11:42.271521 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:42.271072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.66:58603)
0808 02:11:42.271365 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:42.271422 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260808 02:11:42.459051 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 28109
I20260808 02:11:42.467693 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:42.467246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:34709 (local address 127.5.209.67:52569)
0808 02:11:42.467530 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:42.467591 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.126:34709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260808 02:11:42.473034 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 28326
I20260808 02:11:42.484025 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 28504
2026-08-08T02:11:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60156 ms)
[----------] 30 tests from SecurityITest (252361 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:42 dist-test-slave-bw88 krb5kdc[28754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:42 dist-test-slave-bw88 krb5kdc[28754](info): set up 2 sockets
Aug 08 02:11:42 dist-test-slave-bw88 krb5kdc[28754](info): commencing operation
krb5kdc: starting...
W20260808 02:11:43.710194 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:43 dist-test-slave-bw88 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:11:43.848227 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43113
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:43113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:43.964702 28774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:43.964993 28774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:43.965075 28774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:43.969130 28774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:43.969218 28774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:43.969256 28774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:43.969313 28774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:43.969367 28774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:43.974673 28774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:43113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:43.975752 28774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:43.976883 28774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:43.986223 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:43.986315 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:43.986300 28774 server_base.cc:1034] running on GCE node
W20260808 02:11:43.986246 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:43.987133 28774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:43.988159 28774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:43.989369 28774 hybrid_clock.cc:648] HybridClock initialized: now 1786155103989293 us; error 49 us; skew 500 ppm
Aug 08 02:11:43 dist-test-slave-bw88 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155103, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:43.992319 28774 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:11:43.993180 28774 webserver.cc:460] Webserver started at http://127.5.209.126:45843/ using document root <none> and password file <none>
I20260808 02:11:43.993790 28774 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:43.993854 28774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:43.994030 28774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:43.996136 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "8ae890328090477489ed8f614c2df7be"
format_stamp: "Formatted at 2026-08-08 02:11:43 on dist-test-slave-bw88"
server_key: "8681dbb968fd5e96d6a717ec31940c34"
server_key_iv: "82ff836a6385c4c81d2af1314b9ff7f8"
server_key_version: "encryptionkey@0"
I20260808 02:11:43.996672 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "8ae890328090477489ed8f614c2df7be"
format_stamp: "Formatted at 2026-08-08 02:11:43 on dist-test-slave-bw88"
server_key: "8681dbb968fd5e96d6a717ec31940c34"
server_key_iv: "82ff836a6385c4c81d2af1314b9ff7f8"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.000500 28774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:11:44.003141 28790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.004638 28774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:44.004747 28774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "8ae890328090477489ed8f614c2df7be"
format_stamp: "Formatted at 2026-08-08 02:11:43 on dist-test-slave-bw88"
server_key: "8681dbb968fd5e96d6a717ec31940c34"
server_key_iv: "82ff836a6385c4c81d2af1314b9ff7f8"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.004839 28774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:44.017776 28774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:44.020805 28774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:44.020990 28774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:44.028790 28774 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:43113
I20260808 02:11:44.028796 28859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:43113 every 8 connection(s)
I20260808 02:11:44.029882 28774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:44.032944 28860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:44.035880 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28774
I20260808 02:11:44.036046 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:44.036376 5957 external_mini_cluster.cc:1428] Setting key acabf19342d774bcfc8d3dc61bbe261e
I20260808 02:11:44.039357 28860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: Bootstrap starting.
I20260808 02:11:44.042197 28860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:44.043074 28860 log.cc:824] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:44.045238 28860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: No bootstrap required, opened a new log
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:44.048611 28860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } }
I20260808 02:11:44.048892 28860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:44.048977 28860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae890328090477489ed8f614c2df7be, State: Initialized, Role: FOLLOWER
I20260808 02:11:44.049505 28860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } }
I20260808 02:11:44.049670 28860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:44.049743 28860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:44.049845 28860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:44.050737 28860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } }
I20260808 02:11:44.051090 28860 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae890328090477489ed8f614c2df7be; no voters:
I20260808 02:11:44.051448 28860 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:44.051527 28868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:44.051841 28868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [term 1 LEADER]: Becoming Leader. State: Replica: 8ae890328090477489ed8f614c2df7be, State: Running, Role: LEADER
I20260808 02:11:44.052210 28868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } }
I20260808 02:11:44.052623 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.038025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.5.209.126:43113)
0808 02:11:44.038479 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.038499 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.038543 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:11:44.039372 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.039399 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.039837 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.040203 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:44.040950 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.042512 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.043542 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.044035 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.044216 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.046866 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:44.046871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:44.046883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:44.046911 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:44.049414 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.049902 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.049915 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.049920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.050081 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.050380 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.050383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.050385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.050804 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:44.051041 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.051363 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.051701 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260808 02:11:44.052886 28860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:44.053951 28870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae890328090477489ed8f614c2df7be. Latest consensus state: current_term: 1 leader_uuid: "8ae890328090477489ed8f614c2df7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } } }
I20260808 02:11:44.054104 28870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:44.054368 28870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae890328090477489ed8f614c2df7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae890328090477489ed8f614c2df7be" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43113 } } }
I20260808 02:11:44.054446 28870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:44.054526 28878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:44.057786 28878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:44.063445 28878 catalog_manager.cc:1347] Generated new cluster ID: dcb2acc4f7b44024948768cd168f3d36
I20260808 02:11:44.063526 28878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:44.073388 28878 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:44.074352 28878 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:44.085162 28878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: Generated new TSK 0
I20260808 02:11:44.085929 28878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:11:44.122740 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:44.247756 28897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:44.248023 28897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:44.248067 28897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:44.252099 28897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:44.252156 28897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:44.252247 28897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:44.257869 28897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:44.259033 28897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:44.260109 28897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:44.269820 28904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:44.269959 28903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:44.269965 28897 server_base.cc:1034] running on GCE node
W20260808 02:11:44.269832 28906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:44.270545 28897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:44.271239 28897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:44.272432 28897 hybrid_clock.cc:648] HybridClock initialized: now 1786155104272414 us; error 30 us; skew 500 ppm
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:44.275249 28897 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:11:44.276078 28897 webserver.cc:460] Webserver started at http://127.5.209.65:45067/ using document root <none> and password file <none>
I20260808 02:11:44.276624 28897 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:44.276686 28897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:44.276861 28897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:44.278966 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "86648fc8447d4b97a15bad74302d4708"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "cdbc233fd4dfdb1d115fe879048b29f6"
server_key_iv: "c1aabbf1a1f2c85634b935781bfef04b"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.279492 28897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "86648fc8447d4b97a15bad74302d4708"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "cdbc233fd4dfdb1d115fe879048b29f6"
server_key_iv: "c1aabbf1a1f2c85634b935781bfef04b"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.283425 28897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 02:11:44.285832 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.287074 28897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:44.287235 28897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "86648fc8447d4b97a15bad74302d4708"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "cdbc233fd4dfdb1d115fe879048b29f6"
server_key_iv: "c1aabbf1a1f2c85634b935781bfef04b"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.287338 28897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:44.301837 28897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:44.304826 28897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:44.305033 28897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:44.305667 28897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:44.306519 28897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:44.306566 28897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.306639 28897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:44.306671 28897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.316221 28897 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:33225
I20260808 02:11:44.316255 29060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:33225 every 8 connection(s)
I20260808 02:11:44.317363 28897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:44.320942 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 28897
I20260808 02:11:44.321099 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:44.321501 5957 external_mini_cluster.cc:1428] Setting key e7960915fef5f1373b75c2532ea103dc
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:44.331894 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.319317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:56039 (local address 127.5.209.126:43113)
0808 02:11:44.319486 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.319490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.320381 (+ 891us) server_negotiation.cc:407] Connection header received
0808 02:11:44.321353 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.321357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.321420 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.321532 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:44.323067 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.323832 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.324986 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.325378 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.325495 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.328210 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:44.328213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:44.328215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:44.328238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:44.329800 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.330330 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.330334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.330336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.330419 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.330773 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.330778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.330779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.330963 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:44.331070 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.331576 (+ 506us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.331716 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:11:44.332726 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.319750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43113 (local address 127.5.209.65:56039)
0808 02:11:44.320225 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.320268 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.321092 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.321729 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.321736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.322214 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:44.322893 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.322916 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.323966 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.323969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.324851 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.324861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.325550 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.325554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.325730 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.326334 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:44.326351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:44.328039 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:44.329924 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.329929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.329944 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.330220 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.330522 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.330525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.330527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.330673 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.331086 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:44.331095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:44.331343 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.331532 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.331854 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260808 02:11:44.334052 29061 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43113
I20260808 02:11:44.334275 29061 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:44.334820 29061 heartbeater.cc:511] Master 127.5.209.126:43113 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
I20260808 02:11:44.336464 28810 ts_manager.cc:194] Registered new tserver with Master: 86648fc8447d4b97a15bad74302d4708 (127.5.209.65:33225)
I20260808 02:11:44.338011 28810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:56039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:11:44.352972 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:44.469846 29067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:44.470086 29067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:44.470129 29067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:44.474107 29067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:44.474161 29067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:44.474246 29067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:44.479523 29067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:44.480592 29067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:44.481696 29067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:44.491531 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:44.491758 29067 server_base.cc:1034] running on GCE node
W20260808 02:11:44.491564 29076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:44.491576 29074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:44.493624 29067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:44.494285 29067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:44.495477 29067 hybrid_clock.cc:648] HybridClock initialized: now 1786155104495444 us; error 46 us; skew 500 ppm
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:44.498286 29067 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:11:44.499155 29067 webserver.cc:460] Webserver started at http://127.5.209.66:43419/ using document root <none> and password file <none>
I20260808 02:11:44.499770 29067 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:44.499866 29067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:44.500124 29067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:44.502306 29067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "8e26bd1b1d784ec1aceebd2f0683d414"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "8c5d6cbb41f0e006821b03b8bde36a3d"
server_key_iv: "0f8cf17c919b1a3f87f9f663184382ed"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.502880 29067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "8e26bd1b1d784ec1aceebd2f0683d414"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "8c5d6cbb41f0e006821b03b8bde36a3d"
server_key_iv: "0f8cf17c919b1a3f87f9f663184382ed"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.506670 29067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:11:44.509115 29084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.510413 29067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:11:44.510556 29067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "8e26bd1b1d784ec1aceebd2f0683d414"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "8c5d6cbb41f0e006821b03b8bde36a3d"
server_key_iv: "0f8cf17c919b1a3f87f9f663184382ed"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.510674 29067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:44.523487 29067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:44.526288 29067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:44.526489 29067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:44.527076 29067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:44.527889 29067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:44.527961 29067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.528028 29067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:44.528087 29067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.537302 29067 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:41937
I20260808 02:11:44.537374 29236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:41937 every 8 connection(s)
I20260808 02:11:44.538209 29067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:44.540014 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29067
I20260808 02:11:44.540163 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:44.540478 5957 external_mini_cluster.cc:1428] Setting key a67746916bdaca2ca831299297c94017
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:11:44.553618 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.540075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:58175 (local address 127.5.209.126:43113)
0808 02:11:44.540232 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.540237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.541257 (+ 1020us) server_negotiation.cc:407] Connection header received
0808 02:11:44.542427 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.542434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.542509 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.542604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:44.543734 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.544507 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.546301 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.546517 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.546600 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.549583 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:44.549590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:44.549596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:44.549631 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:44.551261 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.551828 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.551834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.551838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.551934 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.552375 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.552380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.552383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.552663 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:44.552774 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.553329 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.553472 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:11:44.554685 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.540474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43113 (local address 127.5.209.66:58175)
0808 02:11:44.541050 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.541132 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.542216 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.542768 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.542774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.543129 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:44.543587 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.543600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.544648 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.544655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.546175 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.546191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.546683 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.546690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.546942 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.547607 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:44.547631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:44.549379 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:44.551398 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.551404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.551416 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.551675 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.552076 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.552079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.552082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.552253 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.552779 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:44.552784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:44.553025 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.553196 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.553596 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":146,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:11:44.556121 29237 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43113
I20260808 02:11:44.556411 29237 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:44.557034 29237 heartbeater.cc:511] Master 127.5.209.126:43113 requested a full tablet report, sending...
I20260808 02:11:44.558394 28810 ts_manager.cc:194] Registered new tserver with Master: 8e26bd1b1d784ec1aceebd2f0683d414 (127.5.209.66:41937)
I20260808 02:11:44.559281 28810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:58175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:11:44.573320 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:44.693542 29242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:44.693818 29242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:44.693929 29242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:44.697937 29242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:44.698011 29242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:44.698096 29242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:44.703311 29242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:43113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:44.704348 29242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:44.705524 29242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:44.715606 29251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:44.715656 29249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:44.715667 29248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:44.715842 29242 server_base.cc:1034] running on GCE node
I20260808 02:11:44.716439 29242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:44.717088 29242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:44.718246 29242 hybrid_clock.cc:648] HybridClock initialized: now 1786155104718211 us; error 47 us; skew 500 ppm
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:44.721186 29242 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:11:44.722132 29242 webserver.cc:460] Webserver started at http://127.5.209.67:36443/ using document root <none> and password file <none>
I20260808 02:11:44.722710 29242 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:44.722771 29242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:44.723016 29242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:44.725047 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "a67de00bc8177e50b60465fdc99ee357"
server_key_iv: "6577bff856e45e14509fbc34ee29d51a"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.725656 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "a67de00bc8177e50b60465fdc99ee357"
server_key_iv: "6577bff856e45e14509fbc34ee29d51a"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.729244 29242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:11:44.731761 29260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.732931 29242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:44.733055 29242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd"
format_stamp: "Formatted at 2026-08-08 02:11:44 on dist-test-slave-bw88"
server_key: "a67de00bc8177e50b60465fdc99ee357"
server_key_iv: "6577bff856e45e14509fbc34ee29d51a"
server_key_version: "encryptionkey@0"
I20260808 02:11:44.733158 29242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:44.762426 29242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:44.765974 29242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:44.766194 29242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:44.766865 29242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:44.767730 29242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:44.767779 29242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.767851 29242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:44.767882 29242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:44.777495 29242 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:43345
I20260808 02:11:44.777529 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:43345 every 8 connection(s)
I20260808 02:11:44.778414 29242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:44.781311 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29242
I20260808 02:11:44.781472 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:44.781844 5957 external_mini_cluster.cc:1428] Setting key 8c57ca21e23d547a9c2e4fd7e3b4c97d
Aug 08 02:11:44 dist-test-slave-bw88 krb5kdc[28754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:44.792609 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.780283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:46925 (local address 127.5.209.126:43113)
0808 02:11:44.780439 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.780443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.781774 (+ 1331us) server_negotiation.cc:407] Connection header received
0808 02:11:44.782856 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.782859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.782920 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.783009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:44.784471 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.785013 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.786279 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.786503 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.786618 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.789065 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:44.789070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:44.789073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:44.789102 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:44.790613 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.791144 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.791152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.791153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.791233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.791562 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.791565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.791566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.791739 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:44.791842 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.792365 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.792466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:11:44.793519 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.780675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:43113 (local address 127.5.209.67:46925)
0808 02:11:44.781580 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.781655 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.782611 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.783249 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.783259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.783676 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:44.784326 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.784343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.785159 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.785162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.786134 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.786144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.786669 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.786673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.786846 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.787401 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:44.787418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:44.788895 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:44.790724 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.790728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.790741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.790992 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.791314 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:44.791316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:44.791317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:44.791459 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:44.791856 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:44.791861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:44.792107 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.792314 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.792573 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":120,"threads_started":1}
I20260808 02:11:44.794778 29408 heartbeater.cc:347] Connected to a master server at 127.5.209.126:43113
I20260808 02:11:44.795065 29408 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:44.795583 29408 heartbeater.cc:511] Master 127.5.209.126:43113 requested a full tablet report, sending...
I20260808 02:11:44.796614 28810 ts_manager.cc:194] Registered new tserver with Master: f6a38ccdf1544a21b3e2fa3f4839cfcd (127.5.209.67:43345)
I20260808 02:11:44.797186 28810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:46925
I20260808 02:11:44.807631 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:44.817682 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.809872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.5.209.126:43113)
0808 02:11:44.810008 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.810012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.810216 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:11:44.810364 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.810368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.810416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.810503 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:44.811236 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.811744 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.812649 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.812869 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.812982 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.814433 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:44.814437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:44.814440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:44.814470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:44.815926 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.816361 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.816365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.816367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.816449 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:44.816733 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:44.816737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:44.816739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:44.816969 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:44.817117 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.817383 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.817505 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:11:44.820608 28810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:44.823858 28810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:44.839247 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.832770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:43345 (local address 127.0.0.1:50288)
0808 02:11:44.833362 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.833413 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.833557 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.834418 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.834424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.834447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:44.834634 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.834643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.836064 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.836069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.837442 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.837455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.838619 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.838753 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.838824 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.838932 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.839061 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":93,"threads_started":1}
I20260808 02:11:44.839627 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.832968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.5.209.67:43345)
0808 02:11:44.833843 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.833858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.833886 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:11:44.833971 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.833978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.834219 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.834809 (+ 590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:44.834979 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.835923 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.837585 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.838474 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838587 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.838738 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.839334 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.839468 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":142,"threads_started":1}
I20260808 02:11:44.839646 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.833767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:33225 (local address 127.0.0.1:46668)
0808 02:11:44.834319 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.834335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.834416 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.834988 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.834991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.835012 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:44.835115 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.835122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.836640 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.836643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.837972 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.838000 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.839268 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.839275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.839359 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.839382 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.839465 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.839524 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":163,"thread_start_us":105,"threads_started":1}
I20260808 02:11:44.839874 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.834000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.5.209.66:41937)
0808 02:11:44.834814 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.834833 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.834856 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:11:44.834947 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.834952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.835121 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.835308 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:44.835481 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.836566 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838049 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.838721 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838877 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.839037 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.839181 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.839289 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":124,"threads_started":1}
I20260808 02:11:44.840328 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.833942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.5.209.65:33225)
0808 02:11:44.834495 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.834502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.834529 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:11:44.834613 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.834621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.834831 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.834995 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:44.835265 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.836510 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838125 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.838952 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.839056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.839204 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.839879 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.840172 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":66,"threads_started":1}
I20260808 02:11:44.840581 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.832984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41937 (local address 127.0.0.1:55222)
0808 02:11:44.833507 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.833522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.833638 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.835277 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.835280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.835298 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:44.835382 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.835388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.836683 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.836689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.837916 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.837927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.838910 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.838915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.839029 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.839055 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.839132 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.839205 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":123,"threads_started":1}
I20260808 02:11:44.842900 29325 tablet_service.cc:1467] Processing CreateTablet for tablet deca5871d42a42238058f12c413bbbec (DEFAULT_TABLE table=test-table [id=9c6acb629a1948eeb79256c54dfd6d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:44.843303 28979 tablet_service.cc:1467] Processing CreateTablet for tablet deca5871d42a42238058f12c413bbbec (DEFAULT_TABLE table=test-table [id=9c6acb629a1948eeb79256c54dfd6d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:44.843302 29152 tablet_service.cc:1467] Processing CreateTablet for tablet deca5871d42a42238058f12c413bbbec (DEFAULT_TABLE table=test-table [id=9c6acb629a1948eeb79256c54dfd6d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:44.844074 29325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deca5871d42a42238058f12c413bbbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:44.844367 28979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deca5871d42a42238058f12c413bbbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:44.844373 29152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deca5871d42a42238058f12c413bbbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:44.851060 29430 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Bootstrap starting.
I20260808 02:11:44.852149 29431 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: Bootstrap starting.
I20260808 02:11:44.853717 29432 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Bootstrap starting.
I20260808 02:11:44.854106 29430 tablet_bootstrap.cc:654] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:44.854336 29431 tablet_bootstrap.cc:654] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:44.855275 29431 log.cc:824] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:44.855507 29430 log.cc:824] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:44.856391 29432 tablet_bootstrap.cc:654] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:44.857108 29432 log.cc:824] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:44.857865 29431 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: No bootstrap required, opened a new log
I20260808 02:11:44.858076 29431 ts_tablet_manager.cc:1397] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:44.858721 29430 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: No bootstrap required, opened a new log
I20260808 02:11:44.858911 29430 ts_tablet_manager.cc:1397] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260808 02:11:44.859149 29432 tablet_bootstrap.cc:492] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: No bootstrap required, opened a new log
I20260808 02:11:44.859342 29432 ts_tablet_manager.cc:1397] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:11:44.862375 29432 raft_consensus.cc:348] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.862826 29432 raft_consensus.cc:374] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:44.862943 29432 raft_consensus.cc:729] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e26bd1b1d784ec1aceebd2f0683d414, State: Initialized, Role: FOLLOWER
I20260808 02:11:44.862936 29431 raft_consensus.cc:348] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.862936 29430 raft_consensus.cc:348] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.863214 29430 raft_consensus.cc:374] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:44.863242 29431 raft_consensus.cc:374] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:44.863304 29430 raft_consensus.cc:729] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a38ccdf1544a21b3e2fa3f4839cfcd, State: Initialized, Role: FOLLOWER
I20260808 02:11:44.863332 29431 raft_consensus.cc:729] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86648fc8447d4b97a15bad74302d4708, State: Initialized, Role: FOLLOWER
I20260808 02:11:44.863572 29432 consensus_queue.cc:260] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.863842 29431 consensus_queue.cc:260] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.863842 29430 consensus_queue.cc:260] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.864671 29432 ts_tablet_manager.cc:1428] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 02:11:44.864854 29430 ts_tablet_manager.cc:1428] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 02:11:44.864854 29431 ts_tablet_manager.cc:1428] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260808 02:11:44.865074 29237 heartbeater.cc:503] Master 127.5.209.126:43113 was elected leader, sending a full tablet report...
I20260808 02:11:44.865203 29408 heartbeater.cc:503] Master 127.5.209.126:43113 was elected leader, sending a full tablet report...
I20260808 02:11:44.865203 29061 heartbeater.cc:503] Master 127.5.209.126:43113 was elected leader, sending a full tablet report...
I20260808 02:11:44.866474 29439 raft_consensus.cc:482] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:44.866642 29439 raft_consensus.cc:504] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.868530 29439 leader_election.cc:288] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e26bd1b1d784ec1aceebd2f0683d414 (127.5.209.66:41937), 86648fc8447d4b97a15bad74302d4708 (127.5.209.65:33225)
I20260808 02:11:44.873083 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.868134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:41937 (local address 127.5.209.67:44205)
0808 02:11:44.868407 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.868423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.868518 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.869280 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.869286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.869308 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:44.869470 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.869480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.870525 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.870529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.871780 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.871791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.872614 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.872618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.872731 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.872761 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.872851 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.872927 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260808 02:11:44.873096 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.868703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:44205 (local address 127.5.209.66:41937)
0808 02:11:44.868949 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.868954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.868968 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:11:44.869013 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.869016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.869062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.869159 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:44.869603 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.870391 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.871923 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.872446 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.872574 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.872626 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.872884 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.872972 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260808 02:11:44.873667 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.868134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:33225 (local address 127.5.209.67:45421)
0808 02:11:44.869062 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.869076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:44.869244 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:44.869653 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:44.869656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:44.869674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:44.869801 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.869808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.870991 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.870994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.872128 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:44.872139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.873193 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.873199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:44.873302 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.873327 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:11:44.873426 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:11:44.873504 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":561,"spinlock_wait_cycles":7424,"thread_start_us":112,"threads_started":1}
I20260808 02:11:44.873667 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.868316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:45421 (local address 127.5.209.65:33225)
0808 02:11:44.868424 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.868430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.869151 (+ 721us) server_negotiation.cc:407] Connection header received
0808 02:11:44.869397 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.869400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.869450 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.869498 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:44.869929 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.870844 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.872262 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.872933 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.873057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.873101 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.873448 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.873529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:11:44.873880 29178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deca5871d42a42238058f12c413bbbec" candidate_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" is_pre_election: true
I20260808 02:11:44.874136 29178 raft_consensus.cc:2454] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a38ccdf1544a21b3e2fa3f4839cfcd in term 0.
I20260808 02:11:44.874568 29264 leader_election.cc:302] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e26bd1b1d784ec1aceebd2f0683d414, f6a38ccdf1544a21b3e2fa3f4839cfcd; no voters:
I20260808 02:11:44.874853 29439 raft_consensus.cc:2781] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:44.874838 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deca5871d42a42238058f12c413bbbec" candidate_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86648fc8447d4b97a15bad74302d4708" is_pre_election: true
I20260808 02:11:44.874950 29439 raft_consensus.cc:482] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:44.874997 29439 raft_consensus.cc:3037] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:44.875104 29009 raft_consensus.cc:2454] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a38ccdf1544a21b3e2fa3f4839cfcd in term 0.
I20260808 02:11:44.876219 29439 raft_consensus.cc:504] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.876533 29439 leader_election.cc:288] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [CANDIDATE]: Term 1 election: Requested vote from peers 8e26bd1b1d784ec1aceebd2f0683d414 (127.5.209.66:41937), 86648fc8447d4b97a15bad74302d4708 (127.5.209.65:33225)
I20260808 02:11:44.876892 29178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deca5871d42a42238058f12c413bbbec" candidate_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e26bd1b1d784ec1aceebd2f0683d414"
I20260808 02:11:44.876995 29178 raft_consensus.cc:3037] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:44.877523 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deca5871d42a42238058f12c413bbbec" candidate_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86648fc8447d4b97a15bad74302d4708"
I20260808 02:11:44.877672 29009 raft_consensus.cc:3037] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:44.878190 29178 raft_consensus.cc:2454] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a38ccdf1544a21b3e2fa3f4839cfcd in term 1.
I20260808 02:11:44.878520 29264 leader_election.cc:302] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e26bd1b1d784ec1aceebd2f0683d414, f6a38ccdf1544a21b3e2fa3f4839cfcd; no voters:
I20260808 02:11:44.878688 29439 raft_consensus.cc:2781] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:44.878810 29439 raft_consensus.cc:686] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 1 LEADER]: Becoming Leader. State: Replica: f6a38ccdf1544a21b3e2fa3f4839cfcd, State: Running, Role: LEADER
I20260808 02:11:44.879033 29009 raft_consensus.cc:2454] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a38ccdf1544a21b3e2fa3f4839cfcd in term 1.
I20260808 02:11:44.879133 29439 consensus_queue.cc:237] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } }
I20260808 02:11:44.882480 28808 catalog_manager.cc:5495] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a38ccdf1544a21b3e2fa3f4839cfcd (127.5.209.67). New cstate: current_term: 1 leader_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a38ccdf1544a21b3e2fa3f4839cfcd" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 43345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 } health_report { overall_health: UNKNOWN } } }
I20260808 02:11:44.902305 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.898622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.5.209.67:43345)
0808 02:11:44.898772 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:44.898777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:44.898792 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:11:44.898912 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:44.898915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:44.898970 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:44.899092 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:44.899337 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.899909 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.900782 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:44.901005 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:44.901127 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:44.901492 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:44.901913 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:44.902060 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:44.902098 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 02:11:44.902158 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:11:44.908529 29178 raft_consensus.cc:1261] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 1 FOLLOWER]: Refusing update from remote peer f6a38ccdf1544a21b3e2fa3f4839cfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:44.908528 29009 raft_consensus.cc:1261] T deca5871d42a42238058f12c413bbbec P 86648fc8447d4b97a15bad74302d4708 [term 1 FOLLOWER]: Refusing update from remote peer f6a38ccdf1544a21b3e2fa3f4839cfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:44.909284 29439 consensus_queue.cc:1035] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e26bd1b1d784ec1aceebd2f0683d414" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:44.909587 29441 consensus_queue.cc:1035] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "86648fc8447d4b97a15bad74302d4708" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:44.917917 29447 mvcc.cc:205] Tried to move back new op lower bound from 7316091309697196032 to 7316091309587902464. Current Snapshot: MvccSnapshot[applied={T|T < 7316091309697196032}]
I20260808 02:11:44.927452 28810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:44.927641 28810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:44.930176 28810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ae890328090477489ed8f614c2df7be: Sending DeleteTablet for 3 replicas of tablet deca5871d42a42238058f12c413bbbec
I20260808 02:11:44.930989 29325 tablet_service.cc:1514] Processing DeleteTablet for tablet deca5871d42a42238058f12c413bbbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:44 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:50288
I20260808 02:11:44.931001 28979 tablet_service.cc:1514] Processing DeleteTablet for tablet deca5871d42a42238058f12c413bbbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:44 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:46668
I20260808 02:11:44.931550 29152 tablet_service.cc:1514] Processing DeleteTablet for tablet deca5871d42a42238058f12c413bbbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:44 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:55222
I20260808 02:11:44.931691 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 28897
I20260808 02:11:44.932083 29463 tablet_replica.cc:330] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: stopping tablet replica
I20260808 02:11:44.932416 29463 raft_consensus.cc:2229] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:44.932747 29463 raft_consensus.cc:2258] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:44.937785 29461 tablet_replica.cc:330] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: stopping tablet replica
I20260808 02:11:44.939078 29463 ts_tablet_manager.cc:1905] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:44.941828 29461 raft_consensus.cc:2229] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:44.942098 29463 ts_tablet_manager.cc:1918] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:44.942484 29463 log.cc:1196] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/deca5871d42a42238058f12c413bbbec
I20260808 02:11:44.942459 29461 raft_consensus.cc:2258] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:44.942960 29463 ts_tablet_manager.cc:1939] T deca5871d42a42238058f12c413bbbec P 8e26bd1b1d784ec1aceebd2f0683d414: Deleting consensus metadata
I20260808 02:11:44.944177 29461 ts_tablet_manager.cc:1905] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:44.944379 28795 catalog_manager.cc:4841] TS 8e26bd1b1d784ec1aceebd2f0683d414 (127.5.209.66:41937): tablet deca5871d42a42238058f12c413bbbec (table test-table [id=9c6acb629a1948eeb79256c54dfd6d8a]) successfully deleted
I20260808 02:11:44.947870 29461 ts_tablet_manager.cc:1918] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:44.948015 29461 log.cc:1196] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/deca5871d42a42238058f12c413bbbec
W20260808 02:11:44.948132 28791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:33225 (error 108)
I20260808 02:11:44.948433 29461 ts_tablet_manager.cc:1939] T deca5871d42a42238058f12c413bbbec P f6a38ccdf1544a21b3e2fa3f4839cfcd: Deleting consensus metadata
I20260808 02:11:44.949288 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:44.948971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:33225 (local address 127.0.0.1:46678)
0808 02:11:44.949121 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:44.949195 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 02:11:44.949442 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29067
W20260808 02:11:44.949586 28791 catalog_manager.cc:4584] TS 86648fc8447d4b97a15bad74302d4708 (127.5.209.65:33225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet deca5871d42a42238058f12c413bbbec: Network error: Client connection negotiation failed: client connection to 127.5.209.65:33225: connect: Connection refused (error 111)
I20260808 02:11:44.961578 28794 catalog_manager.cc:4841] TS f6a38ccdf1544a21b3e2fa3f4839cfcd (127.5.209.67:43345): tablet deca5871d42a42238058f12c413bbbec (table test-table [id=9c6acb629a1948eeb79256c54dfd6d8a]) successfully deleted
I20260808 02:11:44.969100 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29242
I20260808 02:11:44.980581 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 28774
2026-08-08T02:11:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2493 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T02:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:45Z Disabled control of system clock
I20260808 02:11:45.020012 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42177
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:45.139859 29472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:45.140132 29472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:45.140168 29472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:45.144196 29472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:45.144263 29472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:45.144277 29472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:45.144294 29472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:45.144309 29472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:45.149219 29472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:42177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:45.150403 29472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:45.151854 29472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:45.160823 29478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.160921 29472 server_base.cc:1034] running on GCE node
W20260808 02:11:45.160802 29482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.160813 29479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.161863 29472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:45.162981 29472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:45.164191 29472 hybrid_clock.cc:648] HybridClock initialized: now 1786155105164174 us; error 27 us; skew 500 ppm
I20260808 02:11:45.166208 29472 webserver.cc:460] Webserver started at http://127.5.209.126:43711/ using document root <none> and password file <none>
I20260808 02:11:45.166841 29472 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:45.166942 29472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:45.167204 29472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:45.169409 29472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "f93dac61552140f1bb80fdea75481473"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "d0910f3db0f45a85eef222d7cdc5699e"
server_key_iv: "e1dc07027eb7d8a17c55d2128ff37c50"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.170049 29472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "f93dac61552140f1bb80fdea75481473"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "d0910f3db0f45a85eef222d7cdc5699e"
server_key_iv: "e1dc07027eb7d8a17c55d2128ff37c50"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.173815 29472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:45.176394 29488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.177814 29472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:45.177975 29472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "f93dac61552140f1bb80fdea75481473"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "d0910f3db0f45a85eef222d7cdc5699e"
server_key_iv: "e1dc07027eb7d8a17c55d2128ff37c50"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.178103 29472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:45.191057 29472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:45.191828 29472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:45.192054 29472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:45.199558 29472 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:42177
I20260808 02:11:45.199589 29556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:42177 every 8 connection(s)
I20260808 02:11:45.200572 29472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:45.203953 29559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:45.206550 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29472
I20260808 02:11:45.206728 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:45.207038 5957 external_mini_cluster.cc:1428] Setting key fabb25179ade70afc4d808fde7ef43b4
I20260808 02:11:45.210832 29559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: Bootstrap starting.
I20260808 02:11:45.213619 29559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:45.214486 29559 log.cc:824] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:45.217248 29559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: No bootstrap required, opened a new log
I20260808 02:11:45.218209 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.208628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.5.209.126:42177)
0808 02:11:45.209433 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.209466 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.209514 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:11:45.210348 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.210372 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.210891 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.211261 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:45.211722 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.213218 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.214234 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.214964 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.215090 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.216089 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:45.216095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:45.216111 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:45.216607 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:45.216680 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:45.216707 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:45.216813 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.216971 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.217249 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":152,"threads_started":1}
I20260808 02:11:45.220839 29559 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } }
I20260808 02:11:45.221055 29559 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:45.221123 29559 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93dac61552140f1bb80fdea75481473, State: Initialized, Role: FOLLOWER
I20260808 02:11:45.221570 29559 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } }
I20260808 02:11:45.221699 29559 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:45.221740 29559 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:45.221812 29559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:45.222685 29559 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } }
I20260808 02:11:45.222985 29559 leader_election.cc:302] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93dac61552140f1bb80fdea75481473; no voters:
I20260808 02:11:45.223251 29559 leader_election.cc:288] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:45.223371 29565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:45.223651 29565 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [term 1 LEADER]: Becoming Leader. State: Replica: f93dac61552140f1bb80fdea75481473, State: Running, Role: LEADER
I20260808 02:11:45.223975 29565 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } }
I20260808 02:11:45.224498 29559 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:45.225874 29569 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93dac61552140f1bb80fdea75481473. Latest consensus state: current_term: 1 leader_uuid: "f93dac61552140f1bb80fdea75481473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } } }
I20260808 02:11:45.225879 29568 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93dac61552140f1bb80fdea75481473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93dac61552140f1bb80fdea75481473" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42177 } } }
I20260808 02:11:45.226086 29568 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:45.226344 29569 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:45.226764 29577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:45.229756 29577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:45.235102 29577 catalog_manager.cc:1347] Generated new cluster ID: fdbd0b4ee1374d52963ea6066ecabc9a
I20260808 02:11:45.235174 29577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:45.247345 29577 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:45.248623 29577 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:45.258982 29577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: Generated new TSK 0
I20260808 02:11:45.259845 29577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:45.271049 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--builtin_ntp_servers=127.5.209.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:45.392419 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:45.392679 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:45.392717 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:45.396665 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:45.396715 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:45.396802 29589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:45.401597 29589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:45.402642 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:45.404038 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:45.414001 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.414013 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.414248 29589 server_base.cc:1034] running on GCE node
W20260808 02:11:45.414158 29601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.414752 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:45.415382 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:45.416558 29589 hybrid_clock.cc:648] HybridClock initialized: now 1786155105416542 us; error 26 us; skew 500 ppm
I20260808 02:11:45.418644 29589 webserver.cc:460] Webserver started at http://127.5.209.65:40171/ using document root <none> and password file <none>
I20260808 02:11:45.419510 29589 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:45.419618 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:45.419845 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:45.422084 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "862f882d72d442c99889cbbd0662659b"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "271febb33f315a923e0c6b1ef9980422"
server_key_iv: "52a27d6607af25ae2267da2da5094de4"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.422679 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "862f882d72d442c99889cbbd0662659b"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "271febb33f315a923e0c6b1ef9980422"
server_key_iv: "52a27d6607af25ae2267da2da5094de4"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.426563 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:11:45.429155 29609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.430454 29589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:45.430591 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "862f882d72d442c99889cbbd0662659b"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "271febb33f315a923e0c6b1ef9980422"
server_key_iv: "52a27d6607af25ae2267da2da5094de4"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.430706 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:45.454674 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:45.455358 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:45.455559 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:45.456161 29589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:45.457129 29589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:45.457188 29589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.457275 29589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:45.457309 29589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.466538 29589 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:44197
I20260808 02:11:45.466585 29756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:44197 every 8 connection(s)
I20260808 02:11:45.467455 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:45.469187 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29589
I20260808 02:11:45.469317 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:45.469730 5957 external_mini_cluster.cc:1428] Setting key 0d35c199151b70b814264134d3b22e08
I20260808 02:11:45.471968 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--builtin_ntp_servers=127.5.209.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:45.478600 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.469497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:33845 (local address 127.5.209.126:42177)
0808 02:11:45.469671 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.469676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.470990 (+ 1314us) server_negotiation.cc:407] Connection header received
0808 02:11:45.472664 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.472671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.472736 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.472834 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:45.474084 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.474953 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.476131 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.476406 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.476537 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.478003 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:45.478007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:45.478009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:45.478021 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:45.478042 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:45.478050 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:45.478113 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.478408 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.478476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:11:45.479755 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.469947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42177 (local address 127.5.209.65:33845)
0808 02:11:45.470824 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.470878 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.472441 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.473087 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.473097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.473204 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:45.473874 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.473892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.475102 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.475105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.475995 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.476004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.476611 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.476618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.476864 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.477683 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:45.477687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:45.477743 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:45.477830 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:45.478144 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:45.478158 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:45.478168 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.478304 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.478632 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":139,"threads_started":1}
I20260808 02:11:45.481070 29757 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42177
I20260808 02:11:45.481390 29757 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:45.482017 29757 heartbeater.cc:511] Master 127.5.209.126:42177 requested a full tablet report, sending...
I20260808 02:11:45.483579 29513 ts_manager.cc:194] Registered new tserver with Master: 862f882d72d442c99889cbbd0662659b (127.5.209.65:44197)
I20260808 02:11:45.484745 29513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.65:33845
W20260808 02:11:45.591189 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:45.591449 29762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:45.591485 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:45.595504 29762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:45.595564 29762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:45.595649 29762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:45.600369 29762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:45.601464 29762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:45.602867 29762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:45.612931 29771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.612926 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.613011 29773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.613201 29762 server_base.cc:1034] running on GCE node
I20260808 02:11:45.613641 29762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:45.614279 29762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:45.615479 29762 hybrid_clock.cc:648] HybridClock initialized: now 1786155105615461 us; error 31 us; skew 500 ppm
I20260808 02:11:45.617506 29762 webserver.cc:460] Webserver started at http://127.5.209.66:35207/ using document root <none> and password file <none>
I20260808 02:11:45.618103 29762 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:45.618201 29762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:45.618441 29762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:45.620513 29762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "46cdd6c32fd2462f9c54fc27566c2536"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "be879ae686c982471a5dff95e4770105"
server_key_iv: "69cce189bbe928cbcb384092f6d959f0"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.621142 29762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "46cdd6c32fd2462f9c54fc27566c2536"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "be879ae686c982471a5dff95e4770105"
server_key_iv: "69cce189bbe928cbcb384092f6d959f0"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.624902 29762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:45.627620 29779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.628834 29762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 02:11:45.628971 29762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "46cdd6c32fd2462f9c54fc27566c2536"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "be879ae686c982471a5dff95e4770105"
server_key_iv: "69cce189bbe928cbcb384092f6d959f0"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.629099 29762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:45.651845 29762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:45.652618 29762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:45.652837 29762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:45.653535 29762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:45.654580 29762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:45.654655 29762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.654731 29762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:45.654776 29762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.664075 29762 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:43617
I20260808 02:11:45.664100 29922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:43617 every 8 connection(s)
I20260808 02:11:45.665089 29762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:45.671641 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29762
I20260808 02:11:45.671769 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:45.672096 5957 external_mini_cluster.cc:1428] Setting key 94adb0ccace3a86d3077d5bfce5d2b2f
I20260808 02:11:45.674438 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--builtin_ntp_servers=127.5.209.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:45.675390 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.667091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:41143 (local address 127.5.209.126:42177)
0808 02:11:45.667259 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.667263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.668334 (+ 1071us) server_negotiation.cc:407] Connection header received
0808 02:11:45.669962 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.669966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.670041 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.670145 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:45.671085 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.671806 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.673206 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.673544 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.673650 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.674926 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:45.674929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:45.674931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:45.674943 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:45.674962 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:45.674970 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:45.675034 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.675224 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.675278 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:11:45.676651 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.667497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42177 (local address 127.5.209.66:41143)
0808 02:11:45.668177 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.668220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.669732 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.670332 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.670339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.670424 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:45.670926 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.670939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.671950 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.671953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.673084 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.673096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.673724 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.673729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.673930 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.674598 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:45.674602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:45.674664 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:45.674751 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:45.675059 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:45.675072 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:45.675081 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.675218 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.675541 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260808 02:11:45.677953 29923 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42177
I20260808 02:11:45.678233 29923 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:45.678879 29923 heartbeater.cc:511] Master 127.5.209.126:42177 requested a full tablet report, sending...
I20260808 02:11:45.680136 29513 ts_manager.cc:194] Registered new tserver with Master: 46cdd6c32fd2462f9c54fc27566c2536 (127.5.209.66:43617)
I20260808 02:11:45.680732 29513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.66:41143
W20260808 02:11:45.793237 29927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:45.793566 29927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:45.793607 29927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:45.797559 29927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:45.797610 29927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:45.797698 29927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:45.802485 29927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:45.803598 29927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:45.805032 29927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:45.815263 29935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.815269 29933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:45.815268 29932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:45.815580 29927 server_base.cc:1034] running on GCE node
I20260808 02:11:45.816042 29927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:45.816702 29927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:45.817889 29927 hybrid_clock.cc:648] HybridClock initialized: now 1786155105817852 us; error 49 us; skew 500 ppm
I20260808 02:11:45.819780 29927 webserver.cc:460] Webserver started at http://127.5.209.67:41927/ using document root <none> and password file <none>
I20260808 02:11:45.820382 29927 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:45.820479 29927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:45.820734 29927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:45.822901 29927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "138e6967ef9044e395c99063cb7d3161"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "46cc13d40333dd963bb81cf213a604b0"
server_key_iv: "884e75cb5c837282a82825a4c4d4c792"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.823485 29927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "138e6967ef9044e395c99063cb7d3161"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "46cc13d40333dd963bb81cf213a604b0"
server_key_iv: "884e75cb5c837282a82825a4c4d4c792"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.827553 29927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:11:45.830049 29944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.831326 29927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:45.831423 29927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "138e6967ef9044e395c99063cb7d3161"
format_stamp: "Formatted at 2026-08-08 02:11:45 on dist-test-slave-bw88"
server_key: "46cc13d40333dd963bb81cf213a604b0"
server_key_iv: "884e75cb5c837282a82825a4c4d4c792"
server_key_version: "encryptionkey@0"
I20260808 02:11:45.831518 29927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:45.852250 29927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:45.852978 29927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:45.853134 29927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:45.853699 29927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:45.854617 29927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:45.854664 29927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.854703 29927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:45.854719 29927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:45.863948 29927 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:42997
I20260808 02:11:45.863976 30084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:42997 every 8 connection(s)
I20260808 02:11:45.864869 29927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:45.873025 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 29927
I20260808 02:11:45.873178 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:45.873577 5957 external_mini_cluster.cc:1428] Setting key 6ce639fe2919f7bc119236d8398c2e9a
I20260808 02:11:45.875346 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.866792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:46117 (local address 127.5.209.126:42177)
0808 02:11:45.866970 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.866974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.868091 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 02:11:45.869689 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.869693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.869745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.869822 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:45.871195 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.871735 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.872964 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.873248 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.873376 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.874809 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:45.874813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:45.874816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:45.874829 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:45.874854 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:45.874868 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:45.874953 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.875135 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.875203 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:11:45.876358 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.867213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42177 (local address 127.5.209.67:46117)
0808 02:11:45.867932 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.867974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.869433 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.870006 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.870013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.870095 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:45.871001 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.871020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.871880 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.871883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.872818 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.872828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.873459 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.873463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.873691 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.874484 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:45.874488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:45.874555 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:45.874660 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:45.874960 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:45.874972 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:45.874979 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.875131 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.875419 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260808 02:11:45.877478 30085 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42177
I20260808 02:11:45.877729 30085 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:45.878266 30085 heartbeater.cc:511] Master 127.5.209.126:42177 requested a full tablet report, sending...
I20260808 02:11:45.879488 29513 ts_manager.cc:194] Registered new tserver with Master: 138e6967ef9044e395c99063cb7d3161 (127.5.209.67:42997)
I20260808 02:11:45.880097 29513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.67:46117
I20260808 02:11:45.887598 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:45.893069 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.889617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.5.209.126:42177)
0808 02:11:45.889744 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.889748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.889991 (+ 243us) server_negotiation.cc:407] Connection header received
0808 02:11:45.890105 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.890107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.890145 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.890205 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:45.890451 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.890947 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.891725 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.891927 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.892002 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.892723 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:45.892726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:45.892728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:45.892740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:45.892756 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:45.892763 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:45.892821 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.892917 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.892962 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:11:45.895715 29513 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:45.898854 29513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:45.913684 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.908032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.5.209.66:43617)
0808 02:11:45.908898 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.908909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.908928 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:11:45.908993 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.908998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.909137 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.909700 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:45.909838 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.910920 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.912129 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.912915 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.913032 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.913242 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.913464 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.913554 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":259,"threads_started":1}
I20260808 02:11:45.913703 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.907219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:42997 (local address 127.0.0.1:55548)
0808 02:11:45.908522 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.908537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.908625 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.909247 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.909253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.909275 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:45.909492 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.909509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.911047 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.911050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.912394 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.912402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.913251 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.913255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.913415 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.913436 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.913493 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.913553 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":283,"spinlock_wait_cycles":2432,"thread_start_us":78,"threads_started":1}
I20260808 02:11:45.913703 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.907282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:43617 (local address 127.0.0.1:59348)
0808 02:11:45.908292 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.908345 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.908476 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.909288 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.909291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.909319 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:45.909518 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.909525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.911046 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.911050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.911996 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.912006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.913067 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.913074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.913202 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.913256 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.913398 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.913528 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":104,"threads_started":1}
I20260808 02:11:45.914273 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.907569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44197 (local address 127.0.0.1:33286)
0808 02:11:45.908673 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:45.908685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:45.908782 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:45.909440 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:45.909444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:45.909464 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:45.909556 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.909564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.910606 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.910612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.912185 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:45.912197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.913323 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.913328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:45.913986 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.914010 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:11:45.914083 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:11:45.914145 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":602,"thread_start_us":104,"threads_started":1}
I20260808 02:11:45.914587 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.907353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.5.209.67:42997)
0808 02:11:45.908791 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.908805 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.908831 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:11:45.908916 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.908921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.909086 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.909253 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:45.909634 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.910892 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.912523 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.913114 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.914094 (+ 980us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.914281 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.914327 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.914418 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":200,"threads_started":1}
I20260808 02:11:45.914865 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:45.908035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.5.209.65:44197)
0808 02:11:45.908489 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:45.908516 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:45.908800 (+ 284us) server_negotiation.cc:407] Connection header received
0808 02:11:45.908897 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:45.908902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:45.909121 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:45.909281 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:45.909693 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.910479 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.912323 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:45.913105 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:45.913203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:45.913418 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:45.914553 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0808 02:11:45.914688 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":131,"threads_started":1}
I20260808 02:11:45.917043 29840 tablet_service.cc:1467] Processing CreateTablet for tablet d691ef9555544b15a9f35e0162416b45 (DEFAULT_TABLE table=test-table [id=f742e2602d9d4891b2d5743b84141bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:45.917104 30001 tablet_service.cc:1467] Processing CreateTablet for tablet d691ef9555544b15a9f35e0162416b45 (DEFAULT_TABLE table=test-table [id=f742e2602d9d4891b2d5743b84141bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:45.917765 29671 tablet_service.cc:1467] Processing CreateTablet for tablet d691ef9555544b15a9f35e0162416b45 (DEFAULT_TABLE table=test-table [id=f742e2602d9d4891b2d5743b84141bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:45.918175 30001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d691ef9555544b15a9f35e0162416b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:45.918206 29840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d691ef9555544b15a9f35e0162416b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:45.918699 29671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d691ef9555544b15a9f35e0162416b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:45.924383 30113 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Bootstrap starting.
I20260808 02:11:45.925870 30114 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Bootstrap starting.
I20260808 02:11:45.926537 30115 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Bootstrap starting.
I20260808 02:11:45.927585 30113 tablet_bootstrap.cc:654] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:45.928041 30114 tablet_bootstrap.cc:654] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:45.928382 30115 tablet_bootstrap.cc:654] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:45.928561 30113 log.cc:824] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:45.928854 30114 log.cc:824] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:45.929198 30115 log.cc:824] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:45.931003 30114 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: No bootstrap required, opened a new log
I20260808 02:11:45.931145 30113 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: No bootstrap required, opened a new log
I20260808 02:11:45.931221 30114 ts_tablet_manager.cc:1397] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:45.931341 30115 tablet_bootstrap.cc:492] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: No bootstrap required, opened a new log
I20260808 02:11:45.931365 30113 ts_tablet_manager.cc:1397] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260808 02:11:45.931531 30115 ts_tablet_manager.cc:1397] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:11:45.935509 30114 raft_consensus.cc:348] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.935750 30114 raft_consensus.cc:374] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:45.935824 30114 raft_consensus.cc:729] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46cdd6c32fd2462f9c54fc27566c2536, State: Initialized, Role: FOLLOWER
I20260808 02:11:45.935834 30113 raft_consensus.cc:348] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.936184 30113 raft_consensus.cc:374] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:45.936266 30113 raft_consensus.cc:729] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138e6967ef9044e395c99063cb7d3161, State: Initialized, Role: FOLLOWER
I20260808 02:11:45.936435 30114 consensus_queue.cc:260] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.936380 30115 raft_consensus.cc:348] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.936605 30115 raft_consensus.cc:374] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:45.936687 30115 raft_consensus.cc:729] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f882d72d442c99889cbbd0662659b, State: Initialized, Role: FOLLOWER
I20260808 02:11:45.936709 30113 consensus_queue.cc:260] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.937110 30115 consensus_queue.cc:260] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:45.937510 30085 heartbeater.cc:503] Master 127.5.209.126:42177 was elected leader, sending a full tablet report...
I20260808 02:11:45.937528 30114 ts_tablet_manager.cc:1428] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:11:45.937711 30113 ts_tablet_manager.cc:1428] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 02:11:45.937870 29923 heartbeater.cc:503] Master 127.5.209.126:42177 was elected leader, sending a full tablet report...
I20260808 02:11:45.939118 30115 ts_tablet_manager.cc:1428] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 02:11:45.939483 29757 heartbeater.cc:503] Master 127.5.209.126:42177 was elected leader, sending a full tablet report...
W20260808 02:11:45.968711 29758 tablet.cc:2369] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:46.013746 30119 raft_consensus.cc:482] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:46.013963 30119 raft_consensus.cc:504] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:46.015205 30119 leader_election.cc:288] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138e6967ef9044e395c99063cb7d3161 (127.5.209.67:42997), 862f882d72d442c99889cbbd0662659b (127.5.209.65:44197)
I20260808 02:11:46.018977 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.015425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:42997 (local address 127.5.209.66:56247)
0808 02:11:46.015656 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:46.015673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:46.015777 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:46.016040 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:46.016043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:46.016074 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:46.016226 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:46.016232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.016996 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.017000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:46.017858 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:46.017868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.018549 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:46.018660 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:46.018678 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:11:46.018766 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:11:46.018836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"spinlock_wait_cycles":5888}
I20260808 02:11:46.018977 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.015593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:56247 (local address 127.5.209.67:42997)
0808 02:11:46.015771 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:46.015776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:46.015791 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:11:46.015877 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:46.015882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:46.015928 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:46.016027 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:46.016346 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.016876 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018017 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.018419 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018557 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:46.018611 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:46.018785 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 02:11:46.018850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 02:11:46.019345 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.015463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44197 (local address 127.5.209.66:40567)
0808 02:11:46.015952 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:46.015963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:46.016136 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:46.016409 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:46.016412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:46.016429 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:46.016573 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:46.016579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.017301 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.017307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:46.018131 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:46.018141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.018857 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:46.018971 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:46.018989 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:11:46.019100 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:11:46.019185 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"spinlock_wait_cycles":23680,"thread_start_us":200,"threads_started":1}
I20260808 02:11:46.019454 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.015617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:40567 (local address 127.5.209.65:44197)
0808 02:11:46.015762 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:46.015767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:46.016062 (+ 295us) server_negotiation.cc:407] Connection header received
0808 02:11:46.016253 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:46.016261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:46.016304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:46.016394 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:46.016696 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.017184 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018273 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.018686 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.018833 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:46.018882 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:46.019225 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 02:11:46.019295 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:11:46.019737 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d691ef9555544b15a9f35e0162416b45" candidate_uuid: "46cdd6c32fd2462f9c54fc27566c2536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138e6967ef9044e395c99063cb7d3161" is_pre_election: true
I20260808 02:11:46.020098 30027 raft_consensus.cc:2454] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46cdd6c32fd2462f9c54fc27566c2536 in term 0.
I20260808 02:11:46.020181 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d691ef9555544b15a9f35e0162416b45" candidate_uuid: "46cdd6c32fd2462f9c54fc27566c2536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f882d72d442c99889cbbd0662659b" is_pre_election: true
I20260808 02:11:46.020531 29701 raft_consensus.cc:2454] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46cdd6c32fd2462f9c54fc27566c2536 in term 0.
I20260808 02:11:46.020624 29787 leader_election.cc:302] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138e6967ef9044e395c99063cb7d3161, 46cdd6c32fd2462f9c54fc27566c2536; no voters:
I20260808 02:11:46.020941 30119 raft_consensus.cc:2781] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:46.021026 30119 raft_consensus.cc:482] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:46.021059 30119 raft_consensus.cc:3037] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:46.022231 30119 raft_consensus.cc:504] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:46.022624 30119 leader_election.cc:288] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [CANDIDATE]: Term 1 election: Requested vote from peers 138e6967ef9044e395c99063cb7d3161 (127.5.209.67:42997), 862f882d72d442c99889cbbd0662659b (127.5.209.65:44197)
I20260808 02:11:46.022970 30027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d691ef9555544b15a9f35e0162416b45" candidate_uuid: "46cdd6c32fd2462f9c54fc27566c2536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138e6967ef9044e395c99063cb7d3161"
I20260808 02:11:46.023042 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d691ef9555544b15a9f35e0162416b45" candidate_uuid: "46cdd6c32fd2462f9c54fc27566c2536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f882d72d442c99889cbbd0662659b"
I20260808 02:11:46.023124 30027 raft_consensus.cc:3037] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:46.023196 29701 raft_consensus.cc:3037] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:46.024202 30027 raft_consensus.cc:2454] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46cdd6c32fd2462f9c54fc27566c2536 in term 1.
I20260808 02:11:46.024369 29701 raft_consensus.cc:2454] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46cdd6c32fd2462f9c54fc27566c2536 in term 1.
I20260808 02:11:46.024605 29787 leader_election.cc:302] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138e6967ef9044e395c99063cb7d3161, 46cdd6c32fd2462f9c54fc27566c2536; no voters:
I20260808 02:11:46.024786 30119 raft_consensus.cc:2781] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:46.025010 30119 raft_consensus.cc:686] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 1 LEADER]: Becoming Leader. State: Replica: 46cdd6c32fd2462f9c54fc27566c2536, State: Running, Role: LEADER
I20260808 02:11:46.025425 30119 consensus_queue.cc:237] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } }
I20260808 02:11:46.028839 29513 catalog_manager.cc:5495] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46cdd6c32fd2462f9c54fc27566c2536 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "46cdd6c32fd2462f9c54fc27566c2536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46cdd6c32fd2462f9c54fc27566c2536" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 43617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 } health_report { overall_health: UNKNOWN } } }
I20260808 02:11:46.067821 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.064278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.5.209.66:43617)
0808 02:11:46.064418 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:46.064422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:46.064458 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:11:46.064649 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:46.064651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:46.064715 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:46.064813 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:46.065046 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.065576 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.066393 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:46.066615 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:46.066693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:46.067057 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:46.067487 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:46.067598 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:46.067667 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:11:46.067711 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:11:46.074542 29701 raft_consensus.cc:1261] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b [term 1 FOLLOWER]: Refusing update from remote peer 46cdd6c32fd2462f9c54fc27566c2536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:46.074542 30027 raft_consensus.cc:1261] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 1 FOLLOWER]: Refusing update from remote peer 46cdd6c32fd2462f9c54fc27566c2536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:46.075491 30119 consensus_queue.cc:1035] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138e6967ef9044e395c99063cb7d3161" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:46.075780 30123 consensus_queue.cc:1035] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862f882d72d442c99889cbbd0662659b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:46.083559 30127 mvcc.cc:205] Tried to move back new op lower bound from 7316091314471788544 to 7316091314282897408. Current Snapshot: MvccSnapshot[applied={T|T < 7316091314471788544}]
I20260808 02:11:46.086580 30128 mvcc.cc:205] Tried to move back new op lower bound from 7316091314471788544 to 7316091314282897408. Current Snapshot: MvccSnapshot[applied={T|T < 7316091314471788544}]
I20260808 02:11:46.092898 29510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:46.093091 29510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:46.095809 29510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93dac61552140f1bb80fdea75481473: Sending DeleteTablet for 3 replicas of tablet d691ef9555544b15a9f35e0162416b45
I20260808 02:11:46.096628 29840 tablet_service.cc:1514] Processing DeleteTablet for tablet d691ef9555544b15a9f35e0162416b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:46 UTC) from {username='slave'} at 127.0.0.1:59348
I20260808 02:11:46.096729 30001 tablet_service.cc:1514] Processing DeleteTablet for tablet d691ef9555544b15a9f35e0162416b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:46 UTC) from {username='slave'} at 127.0.0.1:55548
I20260808 02:11:46.096721 29671 tablet_service.cc:1514] Processing DeleteTablet for tablet d691ef9555544b15a9f35e0162416b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:46 UTC) from {username='slave'} at 127.0.0.1:33286
I20260808 02:11:46.097303 30137 tablet_replica.cc:330] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: stopping tablet replica
I20260808 02:11:46.097451 30139 tablet_replica.cc:330] T d691ef9555544b15a9f35e0162416b45 P 862f882d72d442c99889cbbd0662659b: stopping tablet replica
I20260808 02:11:46.097506 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29589
I20260808 02:11:46.097692 30137 raft_consensus.cc:2229] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:46.098065 30137 raft_consensus.cc:2258] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:46.098376 30138 tablet_replica.cc:330] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: stopping tablet replica
I20260808 02:11:46.098639 30138 raft_consensus.cc:2229] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:46.098874 30138 raft_consensus.cc:2258] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:46.102200 30137 ts_tablet_manager.cc:1905] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:46.105311 30137 ts_tablet_manager.cc:1918] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:46.105451 30137 log.cc:1196] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/d691ef9555544b15a9f35e0162416b45
I20260808 02:11:46.105811 30137 ts_tablet_manager.cc:1939] T d691ef9555544b15a9f35e0162416b45 P 46cdd6c32fd2462f9c54fc27566c2536: Deleting consensus metadata
I20260808 02:11:46.107329 29489 catalog_manager.cc:4841] TS 46cdd6c32fd2462f9c54fc27566c2536 (127.5.209.66:43617): tablet d691ef9555544b15a9f35e0162416b45 (table test-table [id=f742e2602d9d4891b2d5743b84141bbd]) successfully deleted
I20260808 02:11:46.108704 30138 ts_tablet_manager.cc:1905] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:46.111574 30138 ts_tablet_manager.cc:1918] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:46.111692 30138 log.cc:1196] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/d691ef9555544b15a9f35e0162416b45
I20260808 02:11:46.112066 30138 ts_tablet_manager.cc:1939] T d691ef9555544b15a9f35e0162416b45 P 138e6967ef9044e395c99063cb7d3161: Deleting consensus metadata
W20260808 02:11:46.113114 29493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:44197 (error 108)
I20260808 02:11:46.113441 29494 catalog_manager.cc:4841] TS 138e6967ef9044e395c99063cb7d3161 (127.5.209.67:42997): tablet d691ef9555544b15a9f35e0162416b45 (table test-table [id=f742e2602d9d4891b2d5743b84141bbd]) successfully deleted
I20260808 02:11:46.114081 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29762
I20260808 02:11:46.114420 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:46.114081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44197 (local address 127.0.0.1:33300)
0808 02:11:46.114247 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:46.114330 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:44197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 02:11:46.117631 29493 catalog_manager.cc:4584] TS 862f882d72d442c99889cbbd0662659b (127.5.209.65:44197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d691ef9555544b15a9f35e0162416b45: Network error: Client connection negotiation failed: client connection to 127.5.209.65:44197: connect: Connection refused (error 111)
I20260808 02:11:46.126012 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29927
I20260808 02:11:46.138568 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 29472
2026-08-08T02:11:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1155 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:46 dist-test-slave-bw88 krb5kdc[30144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:46 dist-test-slave-bw88 krb5kdc[30144](info): set up 2 sockets
Aug 08 02:11:46 dist-test-slave-bw88 krb5kdc[30144](info): commencing operation
krb5kdc: starting...
W20260808 02:11:47.356592 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:47 dist-test-slave-bw88 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:11:47.488345 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:45539
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:45539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:47.603284 30166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:47.603626 30166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:47.603715 30166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:47.607758 30166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:47.607867 30166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:47.607908 30166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:47.607946 30166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:47.607990 30166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:47.613695 30166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:45539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:45539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:47.614923 30166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:47.616097 30166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:47.625157 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:47.625157 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:47.625228 30172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:47.625527 30166 server_base.cc:1034] running on GCE node
I20260808 02:11:47.626251 30166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:47.627555 30166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:47.628784 30166 hybrid_clock.cc:648] HybridClock initialized: now 1786155107628759 us; error 47 us; skew 500 ppm
Aug 08 02:11:47 dist-test-slave-bw88 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:47.636971 30166 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:11:47.638094 30166 webserver.cc:460] Webserver started at http://127.5.209.126:38225/ using document root <none> and password file <none>
I20260808 02:11:47.638747 30166 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:47.638850 30166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:47.639119 30166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:47.641484 30166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "bbc3d332cd63431696071d7275d5b08c"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "01fdd3725185c071f2d6059bb7a9fec4"
server_key_iv: "7a72d55b9d0c3ca4610bdcd619920b31"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.642127 30166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "bbc3d332cd63431696071d7275d5b08c"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "01fdd3725185c071f2d6059bb7a9fec4"
server_key_iv: "7a72d55b9d0c3ca4610bdcd619920b31"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.646236 30166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:11:47.648906 30184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:47.650326 30166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:47.650494 30166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "bbc3d332cd63431696071d7275d5b08c"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "01fdd3725185c071f2d6059bb7a9fec4"
server_key_iv: "7a72d55b9d0c3ca4610bdcd619920b31"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.650622 30166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:47.670645 30166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:47.675211 30166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:47.675454 30166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:47.683763 30166 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:45539
I20260808 02:11:47.683777 30251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:45539 every 8 connection(s)
I20260808 02:11:47.684841 30166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:47.686084 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30166
I20260808 02:11:47.686231 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:47.686570 5957 external_mini_cluster.cc:1428] Setting key 2bd7f9587bafea5bd8fc2fb19d83d4ee
I20260808 02:11:47.688514 30252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:47.695286 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: Bootstrap starting.
Aug 08 02:11:47 dist-test-slave-bw88 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:47.697849 30252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:47.698601 30252 log.cc:824] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:47.700758 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: No bootstrap required, opened a new log
I20260808 02:11:47.701907 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:47.688088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.5.209.126:45539)
0808 02:11:47.688543 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:47.688587 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:47.688632 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:11:47.689317 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:47.689386 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:47.689742 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:47.690033 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:47.691024 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.692161 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.693470 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.693994 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.694127 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:47.697072 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:47.697077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:47.697090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:47.697117 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:47.699003 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:47.699439 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:47.699448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:47.699457 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:47.699594 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:47.699901 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:47.699904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:47.699906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:47.700308 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:47.700516 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:47.700825 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:11:47.701125 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":155,"threads_started":1}
I20260808 02:11:47.703896 30252 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } }
I20260808 02:11:47.704142 30252 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:47.704268 30252 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc3d332cd63431696071d7275d5b08c, State: Initialized, Role: FOLLOWER
I20260808 02:11:47.704720 30252 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } }
I20260808 02:11:47.704883 30252 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:47.704973 30252 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:47.705094 30252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:47.706037 30252 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } }
I20260808 02:11:47.706380 30252 leader_election.cc:302] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbc3d332cd63431696071d7275d5b08c; no voters:
I20260808 02:11:47.706694 30252 leader_election.cc:288] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:47.706879 30257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:47.707158 30257 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [term 1 LEADER]: Becoming Leader. State: Replica: bbc3d332cd63431696071d7275d5b08c, State: Running, Role: LEADER
I20260808 02:11:47.707487 30257 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } }
I20260808 02:11:47.707943 30252 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:47.709738 30259 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbc3d332cd63431696071d7275d5b08c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } } }
I20260808 02:11:47.709753 30261 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbc3d332cd63431696071d7275d5b08c. Latest consensus state: current_term: 1 leader_uuid: "bbc3d332cd63431696071d7275d5b08c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc3d332cd63431696071d7275d5b08c" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 45539 } } }
I20260808 02:11:47.709909 30261 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:47.709911 30259 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:47.710291 30277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:47.713766 30277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:47.719635 30277 catalog_manager.cc:1347] Generated new cluster ID: e51cd86dc7c6433a9c53075541fd40fa
I20260808 02:11:47.719704 30277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:47.747404 30277 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:47.748413 30277 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:47.757429 30277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: Generated new TSK 0
I20260808 02:11:47.758188 30277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:11:47.800215 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:47.919425 30289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:47.919734 30289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:47.919809 30289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:47.923887 30289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:47.923976 30289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:47.924070 30289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:47.929368 30289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:47.930454 30289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:47.931555 30289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:47.941779 30296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:47.941824 30297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:47.941779 30300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:47.941884 30289 server_base.cc:1034] running on GCE node
I20260808 02:11:47.942350 30289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:47.943089 30289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:47.944288 30289 hybrid_clock.cc:648] HybridClock initialized: now 1786155107944269 us; error 32 us; skew 500 ppm
Aug 08 02:11:47 dist-test-slave-bw88 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:47.947237 30289 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:11:47.948146 30289 webserver.cc:460] Webserver started at http://127.5.209.65:36639/ using document root <none> and password file <none>
I20260808 02:11:47.948764 30289 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:47.948858 30289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:47.949102 30289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:47.951284 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "b8ea787abb46419eb5924f2010b9534f"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "657b008487e393f9fd796ae74fca5fb7"
server_key_iv: "250fe49509ff0271cefec94c21e70ec2"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.951882 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "b8ea787abb46419eb5924f2010b9534f"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "657b008487e393f9fd796ae74fca5fb7"
server_key_iv: "250fe49509ff0271cefec94c21e70ec2"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.955965 30289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 02:11:47.958652 30312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:47.959954 30289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:11:47.960098 30289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "b8ea787abb46419eb5924f2010b9534f"
format_stamp: "Formatted at 2026-08-08 02:11:47 on dist-test-slave-bw88"
server_key: "657b008487e393f9fd796ae74fca5fb7"
server_key_iv: "250fe49509ff0271cefec94c21e70ec2"
server_key_version: "encryptionkey@0"
I20260808 02:11:47.960215 30289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:47.975520 30289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:47.978801 30289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:47.979017 30289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:47.979666 30289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:47.980556 30289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:47.980634 30289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:47.980708 30289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:47.980758 30289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:47.990250 30289 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:41025
I20260808 02:11:47.990279 30453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:41025 every 8 connection(s)
I20260808 02:11:47.991202 30289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:47.998339 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30289
I20260808 02:11:47.998517 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:47.998924 5957 external_mini_cluster.cc:1428] Setting key 4f512aaeadc9b9d3d75340cd65e0759d
Aug 08 02:11:47 dist-test-slave-bw88 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155107, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:48.006309 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:47.993109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:46689 (local address 127.5.209.126:45539)
0808 02:11:47.993257 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:47.993262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:47.994243 (+ 981us) server_negotiation.cc:407] Connection header received
0808 02:11:47.995143 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:47.995146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:47.995194 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:47.995272 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:47.996626 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.997166 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.998782 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.999069 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.999156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.002371 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:48.002374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:48.002376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:48.002401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:48.004018 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.004589 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.004593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.004594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.004677 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.005061 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.005064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.005066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.005263 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:48.005412 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.005966 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.006123 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:11:48.007167 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:47.993589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45539 (local address 127.5.209.65:46689)
0808 02:11:47.994075 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:47.994120 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:47.994920 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:47.995508 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:47.995516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:47.995910 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:47.996471 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:47.996488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.997296 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.997300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:47.998649 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:47.998663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:47.999246 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:47.999251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:47.999488 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.000136 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:48.000153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:48.002192 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:48.004149 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.004155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.004173 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.004456 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.004786 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.004789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.004791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.004945 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.005410 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:48.005417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:48.005704 (+ 287us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.005907 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.006268 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260808 02:11:48.008637 30454 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45539
I20260808 02:11:48.008859 30454 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:48.009596 30454 heartbeater.cc:511] Master 127.5.209.126:45539 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:11:48.011559 30204 ts_manager.cc:194] Registered new tserver with Master: b8ea787abb46419eb5924f2010b9534f (127.5.209.65:41025)
I20260808 02:11:48.012740 30204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:46689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:11:48.030385 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:48.152212 30461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:48.152566 30461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:48.152675 30461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:48.157073 30461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:48.157166 30461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:48.157253 30461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:48.162817 30461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:48.164022 30461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:48.165179 30461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:48.175527 30469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:48.175540 30468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:48.175655 30461 server_base.cc:1034] running on GCE node
W20260808 02:11:48.175540 30473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:48.176306 30461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:48.176970 30461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:48.178144 30461 hybrid_clock.cc:648] HybridClock initialized: now 1786155108178106 us; error 48 us; skew 500 ppm
Aug 08 02:11:48 dist-test-slave-bw88 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:48.181107 30461 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:11:48.182085 30461 webserver.cc:460] Webserver started at http://127.5.209.66:41109/ using document root <none> and password file <none>
I20260808 02:11:48.182691 30461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:48.182787 30461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:48.183051 30461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:48.185207 30461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "7f6d9e791c394649b9544e4e599e6365"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "b0cacca433e4f0ace13eafbd322b4fa6"
server_key_iv: "d4e6b2a8af4563d32c36bf3088a181d9"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.185907 30461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "7f6d9e791c394649b9544e4e599e6365"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "b0cacca433e4f0ace13eafbd322b4fa6"
server_key_iv: "d4e6b2a8af4563d32c36bf3088a181d9"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.189980 30461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:11:48.192483 30481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.193887 30461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:48.194038 30461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "7f6d9e791c394649b9544e4e599e6365"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "b0cacca433e4f0ace13eafbd322b4fa6"
server_key_iv: "d4e6b2a8af4563d32c36bf3088a181d9"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.194159 30461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:48.213622 30461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:48.216934 30461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:48.217173 30461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:48.217972 30461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:48.218874 30461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:48.218950 30461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.219027 30461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:48.219058 30461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.228394 30461 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:35125
I20260808 02:11:48.228418 30638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:35125 every 8 connection(s)
I20260808 02:11:48.229472 30461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:48.238329 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30461
I20260808 02:11:48.238479 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:48.238857 5957 external_mini_cluster.cc:1428] Setting key 9ae0e68e19ceda86cb1485971801658c
Aug 08 02:11:48 dist-test-slave-bw88 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:48.244238 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.231390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:52661 (local address 127.5.209.126:45539)
0808 02:11:48.231542 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.231547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.232477 (+ 930us) server_negotiation.cc:407] Connection header received
0808 02:11:48.233495 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.233499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.233561 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.233667 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:48.234925 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.235470 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.236679 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.236908 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.237045 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.240099 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:48.240103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:48.240105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:48.240129 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:48.241737 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.242395 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.242398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.242399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.242476 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.242896 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.242899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.242900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.243169 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:48.243286 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.243842 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.244020 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 02:11:48.245002 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.231797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45539 (local address 127.5.209.66:52661)
0808 02:11:48.232300 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.232344 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.233227 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.233865 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.233872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.234262 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:48.234771 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.234785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.235607 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.235611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.236547 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.236557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.237111 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.237116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.237290 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.237922 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:48.237938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:48.239912 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:48.241860 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.241870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.241890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.242294 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.242585 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.242591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.242596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.242798 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.243305 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:48.243314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:48.243579 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.243758 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.244030 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260808 02:11:48.246296 30640 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45539
I20260808 02:11:48.246587 30640 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:48.247258 30640 heartbeater.cc:511] Master 127.5.209.126:45539 requested a full tablet report, sending...
I20260808 02:11:48.248557 30204 ts_manager.cc:194] Registered new tserver with Master: 7f6d9e791c394649b9544e4e599e6365 (127.5.209.66:35125)
I20260808 02:11:48.249212 30204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:52661
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:11:48.272308 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:48.393795 30647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:48.394075 30647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:48.394119 30647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:48.398219 30647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:48.398301 30647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:48.398409 30647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:48.403770 30647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:45539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:48.404902 30647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:48.406042 30647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:48.416517 30662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:48.416472 30657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:48.416483 30659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:48.416664 30647 server_base.cc:1034] running on GCE node
I20260808 02:11:48.417193 30647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:48.417899 30647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:48.419099 30647 hybrid_clock.cc:648] HybridClock initialized: now 1786155108419083 us; error 26 us; skew 500 ppm
Aug 08 02:11:48 dist-test-slave-bw88 krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:48.422041 30647 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:11:48.422917 30647 webserver.cc:460] Webserver started at http://127.5.209.67:43193/ using document root <none> and password file <none>
I20260808 02:11:48.423489 30647 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:48.423583 30647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:48.423828 30647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:48.425974 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "08557602b8b2465ca7130603874dff82"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "324f4da6615eb290a4eaf5a386e6aef9"
server_key_iv: "bafb4c38b48f8efef15471a546e2a2ea"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.426570 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "08557602b8b2465ca7130603874dff82"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "324f4da6615eb290a4eaf5a386e6aef9"
server_key_iv: "bafb4c38b48f8efef15471a546e2a2ea"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.430403 30647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:11:48.432925 30673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.434322 30647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:11:48.434444 30647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "08557602b8b2465ca7130603874dff82"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "324f4da6615eb290a4eaf5a386e6aef9"
server_key_iv: "bafb4c38b48f8efef15471a546e2a2ea"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.434585 30647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:48.446046 30647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:48.449452 30647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:48.449695 30647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:48.450340 30647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:48.451189 30647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:48.451238 30647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.451280 30647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:48.451296 30647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.460976 30647 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:37147
I20260808 02:11:48.461062 30819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:37147 every 8 connection(s)
I20260808 02:11:48.461973 30647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:48.470248 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30647
I20260808 02:11:48.470404 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:48.470763 5957 external_mini_cluster.cc:1428] Setting key 1865678c4b7498ba8ec0df89accc84d3
Aug 08 02:11:48 dist-test-slave-bw88 krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:48.477319 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.463946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:37181 (local address 127.5.209.126:45539)
0808 02:11:48.464191 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.464198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.465047 (+ 849us) server_negotiation.cc:407] Connection header received
0808 02:11:48.466194 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.466198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.466259 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.466383 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:48.467688 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.468302 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.469694 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.469944 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.470096 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.473147 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:48.473151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:48.473161 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:48.473191 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:48.475002 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.475582 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.475592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.475594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.475676 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.476094 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.476099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.476101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.476306 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:48.476450 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.477004 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.477143 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260808 02:11:48.478229 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.464358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:45539 (local address 127.5.209.67:37181)
0808 02:11:48.464867 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.464913 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.465960 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.466590 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.466597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.466996 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:48.467526 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.467541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.468453 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.468457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.469547 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.469559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.470156 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.470161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.470395 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.471137 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:48.471166 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:48.472964 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:48.475140 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.475147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.475163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.475453 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.475781 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:48.475785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:48.475787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:48.475982 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:48.476439 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:48.476447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:48.476727 (+ 280us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.476950 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.477318 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260808 02:11:48.479482 30820 heartbeater.cc:347] Connected to a master server at 127.5.209.126:45539
I20260808 02:11:48.479764 30820 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:48.480347 30820 heartbeater.cc:511] Master 127.5.209.126:45539 requested a full tablet report, sending...
I20260808 02:11:48.481519 30204 ts_manager.cc:194] Registered new tserver with Master: 08557602b8b2465ca7130603874dff82 (127.5.209.67:37147)
I20260808 02:11:48.482128 30204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:37181
I20260808 02:11:48.485885 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:48.495278 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.487854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.5.209.126:45539)
0808 02:11:48.487972 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.487975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.488151 (+ 176us) server_negotiation.cc:407] Connection header received
0808 02:11:48.488246 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.488249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.488284 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.488351 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:48.489017 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.489601 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.490394 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.490623 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.490701 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.492198 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:48.492201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:48.492203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:48.492223 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:48.493724 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.494207 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.494210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.494211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.494283 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:48.494573 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:48.494576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:48.494577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:48.494760 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:48.494873 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.495088 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.495163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:11:48.498979 30204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:48.502353 30204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:48.517882 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.512452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41025 (local address 127.0.0.1:35460)
0808 02:11:48.512747 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.512760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.512833 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.513632 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.513635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.513656 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:48.513746 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.513754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.515017 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.515023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.516040 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.516054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517252 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.517405 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517454 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.517592 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.517726 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260808 02:11:48.517894 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.512893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.5.209.65:41025)
0808 02:11:48.513213 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.513219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.513238 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:11:48.513303 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.513307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.513488 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.513895 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:48.514048 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.514897 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.516218 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517108 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517250 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517504 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.517626 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.517726 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260808 02:11:48.518244 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.511780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:37147 (local address 127.0.0.1:54322)
0808 02:11:48.512665 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.512680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.512787 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.513771 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.513777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.513798 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:48.513890 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.513896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.515411 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.515415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.516579 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.516588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517849 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.517963 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517983 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.518046 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.518109 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":4,"thread_start_us":119,"threads_started":1}
I20260808 02:11:48.518253 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.511771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.5.209.66:35125)
0808 02:11:48.512492 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.512510 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.512597 (+ 87us) server_negotiation.cc:407] Connection header received
0808 02:11:48.512853 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.512860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.513096 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.513237 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:48.513810 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.514993 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.516553 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517410 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517493 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517645 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.517996 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.518106 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":146,"threads_started":1}
I20260808 02:11:48.518725 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.511936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.5.209.67:37147)
0808 02:11:48.513093 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.513105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.513130 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:11:48.513209 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.513214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.513435 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.513603 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:48.513999 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.515262 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.516712 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517539 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517795 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.518483 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.518586 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":90,"threads_started":1}
I20260808 02:11:48.518970 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.511552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:35125 (local address 127.0.0.1:47250)
0808 02:11:48.512434 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.512486 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.512640 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.513504 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.513510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.513530 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:48.513667 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.513675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.515164 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.515167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.516409 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.516420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.517743 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.517746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.517838 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.517860 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.518742 (+ 882us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.518828 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260808 02:11:48.521744 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 6edfb401139c4abdb0008d19a1752e99 (DEFAULT_TABLE table=test-table [id=b563caf9a2ce4ca98ec09c5bec67153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:48.521680 30373 tablet_service.cc:1467] Processing CreateTablet for tablet 6edfb401139c4abdb0008d19a1752e99 (DEFAULT_TABLE table=test-table [id=b563caf9a2ce4ca98ec09c5bec67153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:48.522014 30546 tablet_service.cc:1467] Processing CreateTablet for tablet 6edfb401139c4abdb0008d19a1752e99 (DEFAULT_TABLE table=test-table [id=b563caf9a2ce4ca98ec09c5bec67153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:48.522819 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edfb401139c4abdb0008d19a1752e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:48.522855 30373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edfb401139c4abdb0008d19a1752e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:48.522980 30546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edfb401139c4abdb0008d19a1752e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:48.529733 30844 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Bootstrap starting.
I20260808 02:11:48.531163 30845 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: Bootstrap starting.
I20260808 02:11:48.531613 30848 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Bootstrap starting.
I20260808 02:11:48.531984 30844 tablet_bootstrap.cc:654] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:48.533044 30844 log.cc:824] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:48.533530 30845 tablet_bootstrap.cc:654] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:48.534277 30848 tablet_bootstrap.cc:654] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:48.534547 30845 log.cc:824] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:48.535226 30848 log.cc:824] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:48.535611 30844 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: No bootstrap required, opened a new log
I20260808 02:11:48.535821 30844 ts_tablet_manager.cc:1397] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 02:11:48.537467 30845 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: No bootstrap required, opened a new log
I20260808 02:11:48.537554 30848 tablet_bootstrap.cc:492] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: No bootstrap required, opened a new log
I20260808 02:11:48.537772 30848 ts_tablet_manager.cc:1397] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:48.537784 30845 ts_tablet_manager.cc:1397] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260808 02:11:48.539953 30844 raft_consensus.cc:348] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.540349 30844 raft_consensus.cc:374] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:48.540459 30844 raft_consensus.cc:729] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08557602b8b2465ca7130603874dff82, State: Initialized, Role: FOLLOWER
I20260808 02:11:48.540966 30844 consensus_queue.cc:260] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.541915 30820 heartbeater.cc:503] Master 127.5.209.126:45539 was elected leader, sending a full tablet report...
I20260808 02:11:48.541836 30848 raft_consensus.cc:348] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.542124 30848 raft_consensus.cc:374] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:48.542181 30848 raft_consensus.cc:729] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6d9e791c394649b9544e4e599e6365, State: Initialized, Role: FOLLOWER
I20260808 02:11:48.542639 30844 ts_tablet_manager.cc:1428] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260808 02:11:48.542675 30848 consensus_queue.cc:260] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.543402 30640 heartbeater.cc:503] Master 127.5.209.126:45539 was elected leader, sending a full tablet report...
I20260808 02:11:48.543387 30845 raft_consensus.cc:348] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.543678 30845 raft_consensus.cc:374] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:48.543743 30845 raft_consensus.cc:729] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ea787abb46419eb5924f2010b9534f, State: Initialized, Role: FOLLOWER
I20260808 02:11:48.544240 30848 ts_tablet_manager.cc:1428] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:11:48.544252 30845 consensus_queue.cc:260] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.545099 30454 heartbeater.cc:503] Master 127.5.209.126:45539 was elected leader, sending a full tablet report...
I20260808 02:11:48.546321 30845 ts_tablet_manager.cc:1428] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260808 02:11:48.628726 30854 raft_consensus.cc:482] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:48.628960 30854 raft_consensus.cc:504] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.630293 30854 leader_election.cc:288] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6d9e791c394649b9544e4e599e6365 (127.5.209.66:35125), 08557602b8b2465ca7130603874dff82 (127.5.209.67:37147)
I20260808 02:11:48.634929 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.630751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:60655 (local address 127.5.209.67:37147)
0808 02:11:48.630883 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.630889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.631192 (+ 303us) server_negotiation.cc:407] Connection header received
0808 02:11:48.631354 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.631357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.631412 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.631535 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:48.631826 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.632328 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.633649 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.634256 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.634387 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.634442 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.634748 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.634825 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:11:48.634948 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.630585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:35125 (local address 127.5.209.65:37729)
0808 02:11:48.630987 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.631001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.631158 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.631433 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.631436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.631453 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:48.631599 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.631605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.632357 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.632360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.633465 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.633475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.634420 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.634424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.634591 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.634609 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.634693 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.634774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":43,"thread_start_us":137,"threads_started":1}
I20260808 02:11:48.635005 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.630676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:37729 (local address 127.5.209.66:35125)
0808 02:11:48.630819 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.630824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.631088 (+ 264us) server_negotiation.cc:407] Connection header received
0808 02:11:48.631275 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.631278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.631333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.631420 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:48.631712 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.632240 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.633595 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.634256 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.634387 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.634442 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.634765 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.634865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:11:48.635356 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.630611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:37147 (local address 127.5.209.65:60655)
0808 02:11:48.631085 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.631097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:48.631232 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:48.631557 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:48.631559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:48.631577 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:48.631700 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.631705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.632445 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.632452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.633481 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:48.633493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.634420 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.634424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:48.634561 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.634591 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:11:48.635105 (+ 514us) client_negotiation.cc:240] Negotiation successful
0808 02:11:48.635193 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":47,"thread_start_us":123,"threads_started":1}
I20260808 02:11:48.636222 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edfb401139c4abdb0008d19a1752e99" candidate_uuid: "b8ea787abb46419eb5924f2010b9534f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08557602b8b2465ca7130603874dff82" is_pre_election: true
I20260808 02:11:48.636580 30760 raft_consensus.cc:2454] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ea787abb46419eb5924f2010b9534f in term 0.
I20260808 02:11:48.637146 30315 leader_election.cc:302] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08557602b8b2465ca7130603874dff82, b8ea787abb46419eb5924f2010b9534f; no voters:
I20260808 02:11:48.637591 30854 raft_consensus.cc:2781] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:48.637701 30854 raft_consensus.cc:482] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:48.637754 30854 raft_consensus.cc:3037] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:48.637830 30569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edfb401139c4abdb0008d19a1752e99" candidate_uuid: "b8ea787abb46419eb5924f2010b9534f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6d9e791c394649b9544e4e599e6365" is_pre_election: true
I20260808 02:11:48.638152 30569 raft_consensus.cc:2454] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ea787abb46419eb5924f2010b9534f in term 0.
I20260808 02:11:48.639181 30854 raft_consensus.cc:504] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.639590 30854 leader_election.cc:288] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [CANDIDATE]: Term 1 election: Requested vote from peers 7f6d9e791c394649b9544e4e599e6365 (127.5.209.66:35125), 08557602b8b2465ca7130603874dff82 (127.5.209.67:37147)
I20260808 02:11:48.639992 30569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edfb401139c4abdb0008d19a1752e99" candidate_uuid: "b8ea787abb46419eb5924f2010b9534f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6d9e791c394649b9544e4e599e6365"
I20260808 02:11:48.640007 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edfb401139c4abdb0008d19a1752e99" candidate_uuid: "b8ea787abb46419eb5924f2010b9534f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08557602b8b2465ca7130603874dff82"
I20260808 02:11:48.640136 30569 raft_consensus.cc:3037] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:48.640199 30760 raft_consensus.cc:3037] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:48.641227 30569 raft_consensus.cc:2454] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ea787abb46419eb5924f2010b9534f in term 1.
I20260808 02:11:48.641527 30760 raft_consensus.cc:2454] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ea787abb46419eb5924f2010b9534f in term 1.
I20260808 02:11:48.641613 30314 leader_election.cc:302] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6d9e791c394649b9544e4e599e6365, b8ea787abb46419eb5924f2010b9534f; no voters:
I20260808 02:11:48.641780 30854 raft_consensus.cc:2781] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:48.642063 30854 raft_consensus.cc:686] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 1 LEADER]: Becoming Leader. State: Replica: b8ea787abb46419eb5924f2010b9534f, State: Running, Role: LEADER
I20260808 02:11:48.642408 30854 consensus_queue.cc:237] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } }
I20260808 02:11:48.645679 30202 catalog_manager.cc:5495] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ea787abb46419eb5924f2010b9534f (127.5.209.65). New cstate: current_term: 1 leader_uuid: "b8ea787abb46419eb5924f2010b9534f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ea787abb46419eb5924f2010b9534f" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41025 } health_report { overall_health: HEALTHY } } }
I20260808 02:11:48.672585 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.668884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.5.209.65:41025)
0808 02:11:48.669036 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.669039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.669055 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:11:48.669168 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.669170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.669228 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.669321 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:48.669662 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.670184 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.671031 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.671246 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.671328 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.671650 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:48.672090 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:48.672207 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.672418 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.672459 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:11:48.679232 30760 raft_consensus.cc:1261] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 1 FOLLOWER]: Refusing update from remote peer b8ea787abb46419eb5924f2010b9534f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:48.679234 30569 raft_consensus.cc:1261] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 1 FOLLOWER]: Refusing update from remote peer b8ea787abb46419eb5924f2010b9534f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:48.680038 30857 consensus_queue.cc:1035] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6d9e791c394649b9544e4e599e6365" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 35125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:48.680295 30854 consensus_queue.cc:1035] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08557602b8b2465ca7130603874dff82" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:48.690259 30861 mvcc.cc:205] Tried to move back new op lower bound from 7316091325141159936 to 7316091325002059776. Current Snapshot: MvccSnapshot[applied={T|T < 7316091325141159936}]
I20260808 02:11:48.699841 30202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:48.700042 30202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:48.702800 30202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbc3d332cd63431696071d7275d5b08c: Sending DeleteTablet for 3 replicas of tablet 6edfb401139c4abdb0008d19a1752e99
I20260808 02:11:48.703768 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edfb401139c4abdb0008d19a1752e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:48 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:54322
I20260808 02:11:48.703781 30373 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edfb401139c4abdb0008d19a1752e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:48 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:35460
I20260808 02:11:48.703846 30546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edfb401139c4abdb0008d19a1752e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:48 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:47250
I20260808 02:11:48.704675 30870 tablet_replica.cc:330] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f: stopping tablet replica
I20260808 02:11:48.704941 30871 tablet_replica.cc:330] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: stopping tablet replica
I20260808 02:11:48.705614 30870 raft_consensus.cc:2229] T 6edfb401139c4abdb0008d19a1752e99 P b8ea787abb46419eb5924f2010b9534f [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:48.705680 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30289
I20260808 02:11:48.705318 30871 raft_consensus.cc:2229] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:48.706413 30871 raft_consensus.cc:2258] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:48.707366 30869 tablet_replica.cc:330] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: stopping tablet replica
I20260808 02:11:48.707667 30869 raft_consensus.cc:2229] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:48.707918 30869 raft_consensus.cc:2258] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:48.710464 30869 ts_tablet_manager.cc:1905] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:48.713972 30869 ts_tablet_manager.cc:1918] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:48.714104 30869 log.cc:1196] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/6edfb401139c4abdb0008d19a1752e99
I20260808 02:11:48.714125 30871 ts_tablet_manager.cc:1905] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:48.714511 30869 ts_tablet_manager.cc:1939] T 6edfb401139c4abdb0008d19a1752e99 P 7f6d9e791c394649b9544e4e599e6365: Deleting consensus metadata
I20260808 02:11:48.715703 30189 catalog_manager.cc:4841] TS 7f6d9e791c394649b9544e4e599e6365 (127.5.209.66:35125): tablet 6edfb401139c4abdb0008d19a1752e99 (table test-table [id=b563caf9a2ce4ca98ec09c5bec67153d]) successfully deleted
I20260808 02:11:48.718304 30871 ts_tablet_manager.cc:1918] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:48.718420 30871 log.cc:1196] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/6edfb401139c4abdb0008d19a1752e99
I20260808 02:11:48.718787 30871 ts_tablet_manager.cc:1939] T 6edfb401139c4abdb0008d19a1752e99 P 08557602b8b2465ca7130603874dff82: Deleting consensus metadata
I20260808 02:11:48.720196 30190 catalog_manager.cc:4841] TS 08557602b8b2465ca7130603874dff82 (127.5.209.67:37147): tablet 6edfb401139c4abdb0008d19a1752e99 (table test-table [id=b563caf9a2ce4ca98ec09c5bec67153d]) successfully deleted
W20260808 02:11:48.721000 30191 connection.cc:535] server connection from 127.5.209.65:46689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 02:11:48.721091 30189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:41025 (error 108)
I20260808 02:11:48.722043 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30461
I20260808 02:11:48.722246 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.721905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41025 (local address 127.0.0.1:35482)
0808 02:11:48.722057 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:48.722142 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:11:48.729736 30189 catalog_manager.cc:4584] TS b8ea787abb46419eb5924f2010b9534f (127.5.209.65:41025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6edfb401139c4abdb0008d19a1752e99: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41025: connect: Connection refused (error 111)
I20260808 02:11:48.741611 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30647
I20260808 02:11:48.753436 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30166
2026-08-08T02:11:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2619 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T02:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:48Z Disabled control of system clock
I20260808 02:11:48.794690 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42641
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:48.910295 30877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:48.910568 30877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:48.910604 30877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:48.914580 30877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:48.914639 30877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:48.914655 30877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:48.914671 30877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:48.914686 30877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:48.919456 30877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:42641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:48.920529 30877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:48.921958 30877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:48.930765 30883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:48.930819 30877 server_base.cc:1034] running on GCE node
W20260808 02:11:48.930773 30889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:48.930902 30884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:48.931574 30877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:48.932655 30877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:48.933842 30877 hybrid_clock.cc:648] HybridClock initialized: now 1786155108933830 us; error 29 us; skew 500 ppm
I20260808 02:11:48.935765 30877 webserver.cc:460] Webserver started at http://127.5.209.126:44061/ using document root <none> and password file <none>
I20260808 02:11:48.936368 30877 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:48.936462 30877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:48.936709 30877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:48.938848 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "7f714239e71c7121b6a2dd727cead358"
server_key_iv: "6c7202925fe31737ddd3cb0514104131"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.939462 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "7f714239e71c7121b6a2dd727cead358"
server_key_iv: "6c7202925fe31737ddd3cb0514104131"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.943325 30877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:11:48.945847 30895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:48.947090 30877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:48.947242 30877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd"
format_stamp: "Formatted at 2026-08-08 02:11:48 on dist-test-slave-bw88"
server_key: "7f714239e71c7121b6a2dd727cead358"
server_key_iv: "6c7202925fe31737ddd3cb0514104131"
server_key_version: "encryptionkey@0"
I20260808 02:11:48.947360 30877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:48.963285 30877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:48.964001 30877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:48.964212 30877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:48.971467 30877 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:42641
I20260808 02:11:48.971513 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:42641 every 8 connection(s)
I20260808 02:11:48.972470 30877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:48.975724 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:48.981065 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30877
I20260808 02:11:48.981196 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:48.981550 5957 external_mini_cluster.cc:1428] Setting key 555b6813cd365b0b9c88f75856c0f972
I20260808 02:11:48.981565 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: Bootstrap starting.
I20260808 02:11:48.984324 30958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:48.985124 30958 log.cc:824] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:48.987340 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: No bootstrap required, opened a new log
I20260808 02:11:48.991169 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:48.982952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.5.209.126:42641)
0808 02:11:48.983738 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:48.983753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:48.983790 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:11:48.984476 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:48.984501 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:48.984978 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:48.985332 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:48.985820 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.986815 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.987773 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:48.988248 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:48.988429 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:48.989381 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:48.989388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:48.989411 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:48.989806 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:48.989863 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:48.989883 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:48.989995 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:48.990170 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 02:11:48.990431 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":176,"threads_started":1}
I20260808 02:11:48.991295 30958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } }
I20260808 02:11:48.991598 30958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:48.991662 30958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bdd7e9bf6ea4562b9e270c5a91cc6dd, State: Initialized, Role: FOLLOWER
I20260808 02:11:48.992139 30958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } }
I20260808 02:11:48.992331 30958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:48.992410 30958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:48.992554 30958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:48.993602 30958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } }
I20260808 02:11:48.993970 30958 leader_election.cc:302] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bdd7e9bf6ea4562b9e270c5a91cc6dd; no voters:
I20260808 02:11:48.994274 30958 leader_election.cc:288] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:48.994413 30965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:48.994630 30965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [term 1 LEADER]: Becoming Leader. State: Replica: 0bdd7e9bf6ea4562b9e270c5a91cc6dd, State: Running, Role: LEADER
I20260808 02:11:48.994976 30965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } }
I20260808 02:11:48.995559 30958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:48.996680 30968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bdd7e9bf6ea4562b9e270c5a91cc6dd. Latest consensus state: current_term: 1 leader_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } } }
I20260808 02:11:48.996819 30968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:48.997238 30978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:48.997195 30967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bdd7e9bf6ea4562b9e270c5a91cc6dd" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42641 } } }
I20260808 02:11:48.997467 30967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:49.000505 30978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:49.006261 30978 catalog_manager.cc:1347] Generated new cluster ID: fc020b22a0c14756a25e0d7b6a916c25
I20260808 02:11:49.006331 30978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:49.020180 30978 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:49.021088 30978 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:49.028470 30978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: Generated new TSK 0
I20260808 02:11:49.029153 30978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:49.035771 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--builtin_ntp_servers=127.5.209.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:49.176080 30990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:49.176375 30990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:49.176470 30990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:49.180541 30990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:49.180632 30990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:49.180723 30990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:49.185729 30990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:49.186865 30990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:49.188333 30990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:49.198566 30998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.198572 30995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.198583 30996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:49.199033 30990 server_base.cc:1034] running on GCE node
I20260808 02:11:49.199414 30990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:49.200083 30990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:49.201246 30990 hybrid_clock.cc:648] HybridClock initialized: now 1786155109201210 us; error 46 us; skew 500 ppm
I20260808 02:11:49.203263 30990 webserver.cc:460] Webserver started at http://127.5.209.65:37485/ using document root <none> and password file <none>
I20260808 02:11:49.203890 30990 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:49.203980 30990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:49.204250 30990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:49.206475 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "d22135282821428eae8471ebea4f910b"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "e43bbce9d450ac6db747a866d9852ea2"
server_key_iv: "431a44c7915118a1d20b1f74025da5f1"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.207069 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "d22135282821428eae8471ebea4f910b"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "e43bbce9d450ac6db747a866d9852ea2"
server_key_iv: "431a44c7915118a1d20b1f74025da5f1"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.210891 30990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:11:49.213503 31007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.214737 30990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:49.214876 30990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "d22135282821428eae8471ebea4f910b"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "e43bbce9d450ac6db747a866d9852ea2"
server_key_iv: "431a44c7915118a1d20b1f74025da5f1"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.214993 30990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:49.230129 30990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:49.230822 30990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:49.231031 30990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:49.231621 30990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:49.232583 30990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:49.232669 30990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.232762 30990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:49.232815 30990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.242861 30990 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:41087
I20260808 02:11:49.242887 31152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:41087 every 8 connection(s)
I20260808 02:11:49.243876 30990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:49.244966 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 30990
I20260808 02:11:49.245106 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:49.245530 5957 external_mini_cluster.cc:1428] Setting key ce1196c3fe7a86479d6d824cf3af0488
I20260808 02:11:49.247810 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--builtin_ntp_servers=127.5.209.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:49.255540 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.246140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:44827 (local address 127.5.209.126:42641)
0808 02:11:49.246301 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.246305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.247613 (+ 1308us) server_negotiation.cc:407] Connection header received
0808 02:11:49.249195 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.249198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.249259 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.249356 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:49.250675 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.251242 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.253120 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.253410 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.253508 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.254917 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:49.254921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:49.254924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:49.254939 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:49.254958 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:49.254966 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:49.255036 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.255302 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.255372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:11:49.256781 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.246615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42641 (local address 127.5.209.65:44827)
0808 02:11:49.247423 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.247505 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.248962 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.249589 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.249601 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.249731 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:49.250487 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.250509 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.251376 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.251382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.252968 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.252984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.253591 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.253598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.253852 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.254563 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:49.254565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:49.254631 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:49.254726 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:49.255071 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:49.255084 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:49.255092 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.255256 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.255617 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":177,"threads_started":1}
I20260808 02:11:49.258380 31153 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42641
I20260808 02:11:49.258661 31153 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:49.259218 31153 heartbeater.cc:511] Master 127.5.209.126:42641 requested a full tablet report, sending...
I20260808 02:11:49.261091 30915 ts_manager.cc:194] Registered new tserver with Master: d22135282821428eae8471ebea4f910b (127.5.209.65:41087)
I20260808 02:11:49.262825 30915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.65:44827
W20260808 02:11:49.368009 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:49.368270 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:49.368315 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:49.372216 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:49.372270 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:49.372321 31157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:49.376919 31157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:49.378000 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:49.379375 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:49.389597 31163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.389724 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.389746 31166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:49.389912 31157 server_base.cc:1034] running on GCE node
I20260808 02:11:49.390400 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:49.391098 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:49.392289 31157 hybrid_clock.cc:648] HybridClock initialized: now 1786155109392252 us; error 49 us; skew 500 ppm
I20260808 02:11:49.394321 31157 webserver.cc:460] Webserver started at http://127.5.209.66:46383/ using document root <none> and password file <none>
I20260808 02:11:49.395048 31157 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:49.395117 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:49.395290 31157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:49.397692 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "f783886cc7034f4f9d2ded8c81fd5ac7"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "1af2af66ae9596c28dcf55d98f30ada2"
server_key_iv: "759cb964bf14004351b1715da3931f80"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.398326 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "f783886cc7034f4f9d2ded8c81fd5ac7"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "1af2af66ae9596c28dcf55d98f30ada2"
server_key_iv: "759cb964bf14004351b1715da3931f80"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.402139 31157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:11:49.404661 31172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.405915 31157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:11:49.406064 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "f783886cc7034f4f9d2ded8c81fd5ac7"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "1af2af66ae9596c28dcf55d98f30ada2"
server_key_iv: "759cb964bf14004351b1715da3931f80"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.406183 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:49.422982 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:49.423707 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:49.423918 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:49.424530 31157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:49.425568 31157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:49.425639 31157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.425711 31157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:49.425751 31157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.434739 31157 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:38969
I20260808 02:11:49.434767 31315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:38969 every 8 connection(s)
I20260808 02:11:49.435678 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:49.436300 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 31157
I20260808 02:11:49.436445 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:49.436825 5957 external_mini_cluster.cc:1428] Setting key 30d8854c84bfbce8a7e57ff3a51a8788
I20260808 02:11:49.439116 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--builtin_ntp_servers=127.5.209.84:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:49.447455 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.437915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:48973 (local address 127.5.209.126:42641)
0808 02:11:49.438061 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.438065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.439542 (+ 1477us) server_negotiation.cc:407] Connection header received
0808 02:11:49.441700 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.441705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.441770 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.441872 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:49.442841 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.443707 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.445035 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.445525 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.445658 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.446844 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:49.446850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:49.446854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:49.446873 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:49.446902 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:49.446917 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:49.447015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.447160 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.447238 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:11:49.448258 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.438538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42641 (local address 127.5.209.66:48973)
0808 02:11:49.439363 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.439433 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.441466 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.442041 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.442048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.442137 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:49.442687 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.442701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.443855 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.443858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.444888 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.444899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.445694 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.445701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.445882 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.446561 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:49.446562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:49.446604 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:49.446673 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:49.447012 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:49.447019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:49.447024 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.447121 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.447359 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":115,"threads_started":1}
I20260808 02:11:49.449733 31316 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42641
I20260808 02:11:49.450037 31316 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:49.450796 31316 heartbeater.cc:511] Master 127.5.209.126:42641 requested a full tablet report, sending...
I20260808 02:11:49.452119 30915 ts_manager.cc:194] Registered new tserver with Master: f783886cc7034f4f9d2ded8c81fd5ac7 (127.5.209.66:38969)
I20260808 02:11:49.452771 30915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.66:48973
W20260808 02:11:49.560696 31319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:49.561007 31319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:49.561084 31319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:49.565114 31319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:49.565203 31319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:49.565297 31319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:49.570164 31319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:49.571324 31319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:49.572820 31319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:49.582688 31327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.582688 31329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:49.582748 31332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:49.582880 31319 server_base.cc:1034] running on GCE node
I20260808 02:11:49.583359 31319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:49.584033 31319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:49.585269 31319 hybrid_clock.cc:648] HybridClock initialized: now 1786155109585252 us; error 28 us; skew 500 ppm
I20260808 02:11:49.587317 31319 webserver.cc:460] Webserver started at http://127.5.209.67:39155/ using document root <none> and password file <none>
I20260808 02:11:49.587942 31319 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:49.588039 31319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:49.588296 31319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:49.590468 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "884ae250358244448cc8ab1931316553"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "212b37074b3ae7e506a8380ac1b71cfd"
server_key_iv: "f017b056417a3628d2ae74692540be9b"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.591053 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "884ae250358244448cc8ab1931316553"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "212b37074b3ae7e506a8380ac1b71cfd"
server_key_iv: "f017b056417a3628d2ae74692540be9b"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.594820 31319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:11:49.597537 31338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.598779 31319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:49.598918 31319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "884ae250358244448cc8ab1931316553"
format_stamp: "Formatted at 2026-08-08 02:11:49 on dist-test-slave-bw88"
server_key: "212b37074b3ae7e506a8380ac1b71cfd"
server_key_iv: "f017b056417a3628d2ae74692540be9b"
server_key_version: "encryptionkey@0"
I20260808 02:11:49.599036 31319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:49.623256 31319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:49.623974 31319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:49.624184 31319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:49.624804 31319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:49.625805 31319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:49.625878 31319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.625953 31319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:49.626008 31319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:49.636160 31319 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:33749
I20260808 02:11:49.636216 31478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:33749 every 8 connection(s)
I20260808 02:11:49.637209 31319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:49.637457 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 31319
I20260808 02:11:49.637580 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:49.637946 5957 external_mini_cluster.cc:1428] Setting key 0b011d2d6110cdcf2c821220eb9d36d7
I20260808 02:11:49.647187 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.639275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:49117 (local address 127.5.209.126:42641)
0808 02:11:49.639440 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.639443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.640819 (+ 1376us) server_negotiation.cc:407] Connection header received
0808 02:11:49.642298 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.642301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.642358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.642437 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:49.643342 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.643893 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.645108 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.645386 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.645506 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.646666 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:49.646669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:49.646672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:49.646686 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:49.646711 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:49.646725 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:49.646805 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.646973 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.647044 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:11:49.648094 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.639874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:42641 (local address 127.5.209.67:49117)
0808 02:11:49.640673 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.640715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.642088 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.642617 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.642624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.642708 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:49.643190 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.643203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.644064 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.644068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.644974 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.644984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.645559 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.645564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.645733 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.646392 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:49.646393 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:49.646435 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:49.646499 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:49.646808 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:49.646821 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:49.646830 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.646971 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.647260 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":161,"threads_started":1}
I20260808 02:11:49.649220 31480 heartbeater.cc:347] Connected to a master server at 127.5.209.126:42641
I20260808 02:11:49.649482 31480 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:49.650025 31480 heartbeater.cc:511] Master 127.5.209.126:42641 requested a full tablet report, sending...
I20260808 02:11:49.651232 30915 ts_manager.cc:194] Registered new tserver with Master: 884ae250358244448cc8ab1931316553 (127.5.209.67:33749)
I20260808 02:11:49.651995 30915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.67:49117
I20260808 02:11:49.652240 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:49.658633 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.654535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.5.209.126:42641)
0808 02:11:49.654695 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.654700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.654869 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:11:49.655000 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.655013 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.655061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.655145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:49.655449 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.655943 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.656823 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.657061 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.657173 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.658128 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:49.658132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:49.658135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:49.658149 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:49.658176 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:49.658184 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:49.658318 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.658449 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.658486 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:11:49.661453 30915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:49.664458 30915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:49.679180 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.673974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41087 (local address 127.0.0.1:55208)
0808 02:11:49.674389 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.674403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.674487 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.675167 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.675170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.675189 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:49.675280 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.675288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676304 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.676309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.677578 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.677589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.678626 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.678629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.678734 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.678770 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.678889 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.679025 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260808 02:11:49.679186 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.674429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.5.209.65:41087)
0808 02:11:49.674778 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.674788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.674809 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:11:49.674876 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.674879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.675021 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.675183 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:49.675418 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676186 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.677704 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.678492 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.678625 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.678833 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.678919 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.679017 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":146,"threads_started":1}
I20260808 02:11:49.679539 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.673280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33749 (local address 127.0.0.1:43044)
0808 02:11:49.673987 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.674001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.674115 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.674931 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.674937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.674961 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:49.675126 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.675138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676710 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.676713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.678133 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.678144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.679122 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.679125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.679240 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.679264 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.679328 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.679393 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":136,"thread_start_us":141,"threads_started":1}
I20260808 02:11:49.679977 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.673057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:38969 (local address 127.0.0.1:36282)
0808 02:11:49.673491 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.673544 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.673697 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.675217 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.675223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.675244 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:49.675325 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.675330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676836 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.676839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.677999 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.678009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.679101 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.679106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.679597 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.679652 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.679727 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.679808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":397,"thread_start_us":101,"threads_started":1}
I20260808 02:11:49.680387 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.673228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.5.209.66:38969)
0808 02:11:49.674742 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.674754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.674779 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:11:49.674858 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.674866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.675052 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.675465 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:49.675616 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676725 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.678130 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.678932 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.679266 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.679460 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.680113 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.680229 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":160,"threads_started":1}
I20260808 02:11:49.680785 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.673937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.5.209.67:33749)
0808 02:11:49.674377 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.674390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.674418 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:11:49.674520 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.674530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.674753 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.675288 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:49.675435 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.676574 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.678261 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.679006 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.680228 (+ 1222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.680417 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.680475 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.680589 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":167,"threads_started":1}
I20260808 02:11:49.683105 31235 tablet_service.cc:1467] Processing CreateTablet for tablet c4b51dabe135457c99b91e194688da01 (DEFAULT_TABLE table=test-table [id=be12613e6c5b48cb8f24599e721985ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:49.683041 31069 tablet_service.cc:1467] Processing CreateTablet for tablet c4b51dabe135457c99b91e194688da01 (DEFAULT_TABLE table=test-table [id=be12613e6c5b48cb8f24599e721985ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:49.683525 31397 tablet_service.cc:1467] Processing CreateTablet for tablet c4b51dabe135457c99b91e194688da01 (DEFAULT_TABLE table=test-table [id=be12613e6c5b48cb8f24599e721985ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:49.684221 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b51dabe135457c99b91e194688da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:49.684221 31235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b51dabe135457c99b91e194688da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:49.684420 31397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b51dabe135457c99b91e194688da01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:49.690757 31506 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Bootstrap starting.
I20260808 02:11:49.692256 31507 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Bootstrap starting.
I20260808 02:11:49.693034 31506 tablet_bootstrap.cc:654] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:49.693691 31509 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Bootstrap starting.
I20260808 02:11:49.694020 31506 log.cc:824] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:49.694211 31507 tablet_bootstrap.cc:654] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:49.695097 31507 log.cc:824] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:49.695864 31509 tablet_bootstrap.cc:654] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:49.696666 31509 log.cc:824] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:49.697359 31506 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: No bootstrap required, opened a new log
I20260808 02:11:49.697567 31506 ts_tablet_manager.cc:1397] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260808 02:11:49.697839 31507 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: No bootstrap required, opened a new log
I20260808 02:11:49.698086 31507 ts_tablet_manager.cc:1397] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:49.698832 31509 tablet_bootstrap.cc:492] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: No bootstrap required, opened a new log
I20260808 02:11:49.699033 31509 ts_tablet_manager.cc:1397] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:11:49.702244 31506 raft_consensus.cc:348] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.702646 31506 raft_consensus.cc:374] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:49.702718 31506 raft_consensus.cc:729] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f783886cc7034f4f9d2ded8c81fd5ac7, State: Initialized, Role: FOLLOWER
I20260808 02:11:49.702455 31509 raft_consensus.cc:348] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.702450 31507 raft_consensus.cc:348] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.702814 31507 raft_consensus.cc:374] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:49.702836 31509 raft_consensus.cc:374] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:49.702872 31507 raft_consensus.cc:729] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22135282821428eae8471ebea4f910b, State: Initialized, Role: FOLLOWER
I20260808 02:11:49.702935 31509 raft_consensus.cc:729] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884ae250358244448cc8ab1931316553, State: Initialized, Role: FOLLOWER
I20260808 02:11:49.703251 31506 consensus_queue.cc:260] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.703310 31509 consensus_queue.cc:260] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.703359 31507 consensus_queue.cc:260] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.704221 31480 heartbeater.cc:503] Master 127.5.209.126:42641 was elected leader, sending a full tablet report...
I20260808 02:11:49.704326 31509 ts_tablet_manager.cc:1428] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 02:11:49.704330 31507 ts_tablet_manager.cc:1428] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:49.704330 31506 ts_tablet_manager.cc:1428] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 02:11:49.704685 31153 heartbeater.cc:503] Master 127.5.209.126:42641 was elected leader, sending a full tablet report...
I20260808 02:11:49.705317 31316 heartbeater.cc:503] Master 127.5.209.126:42641 was elected leader, sending a full tablet report...
I20260808 02:11:49.741807 31516 raft_consensus.cc:482] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:49.742058 31516 raft_consensus.cc:504] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.743345 31516 leader_election.cc:288] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 884ae250358244448cc8ab1931316553 (127.5.209.67:33749), d22135282821428eae8471ebea4f910b (127.5.209.65:41087)
W20260808 02:11:49.745108 31154 tablet.cc:2369] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:49.748844 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.743790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:40215 (local address 127.5.209.65:41087)
0808 02:11:49.743938 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.743942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.744213 (+ 271us) server_negotiation.cc:407] Connection header received
0808 02:11:49.744332 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.744335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.744388 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.744465 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:49.744888 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.745717 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.747507 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.748125 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.748257 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.748309 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.748626 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.748710 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:11:49.748844 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.743710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:55779 (local address 127.5.209.67:33749)
0808 02:11:49.743882 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.743887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.744086 (+ 199us) server_negotiation.cc:407] Connection header received
0808 02:11:49.744307 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.744310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.744362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.744445 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:49.744887 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.745717 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.747507 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.748132 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.748257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.748309 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.748626 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.748722 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:11:49.748864 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.743569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:33749 (local address 127.5.209.66:55779)
0808 02:11:49.743998 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.744011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.744133 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.744491 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.744493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.744524 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:49.744769 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.744775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.745847 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.745850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.747363 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.747376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.748267 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.748269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.748440 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.748486 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.748578 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.748687 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":269,"spinlock_wait_cycles":4352}
I20260808 02:11:49.748874 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.743687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41087 (local address 127.5.209.66:40215)
0808 02:11:49.744119 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.744131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:49.744235 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:49.744491 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:49.744493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:49.744512 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:49.744768 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.744775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.745847 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.745850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.747370 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:49.747376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.748265 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.748269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:49.748437 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.748493 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:11:49.748575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:11:49.748687 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":262,"thread_start_us":181,"threads_started":1}
I20260808 02:11:49.749682 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b51dabe135457c99b91e194688da01" candidate_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22135282821428eae8471ebea4f910b" is_pre_election: true
I20260808 02:11:49.749891 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b51dabe135457c99b91e194688da01" candidate_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884ae250358244448cc8ab1931316553" is_pre_election: true
I20260808 02:11:49.750056 31095 raft_consensus.cc:2454] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f783886cc7034f4f9d2ded8c81fd5ac7 in term 0.
I20260808 02:11:49.750164 31420 raft_consensus.cc:2454] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f783886cc7034f4f9d2ded8c81fd5ac7 in term 0.
I20260808 02:11:49.750639 31177 leader_election.cc:302] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 884ae250358244448cc8ab1931316553, d22135282821428eae8471ebea4f910b, f783886cc7034f4f9d2ded8c81fd5ac7; no voters:
I20260808 02:11:49.750936 31516 raft_consensus.cc:2781] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:49.751041 31516 raft_consensus.cc:482] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:49.751161 31516 raft_consensus.cc:3037] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:49.752405 31516 raft_consensus.cc:504] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.752764 31516 leader_election.cc:288] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [CANDIDATE]: Term 1 election: Requested vote from peers 884ae250358244448cc8ab1931316553 (127.5.209.67:33749), d22135282821428eae8471ebea4f910b (127.5.209.65:41087)
I20260808 02:11:49.753144 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b51dabe135457c99b91e194688da01" candidate_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22135282821428eae8471ebea4f910b"
I20260808 02:11:49.753170 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b51dabe135457c99b91e194688da01" candidate_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884ae250358244448cc8ab1931316553"
I20260808 02:11:49.753283 31095 raft_consensus.cc:3037] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:49.753283 31420 raft_consensus.cc:3037] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:49.754647 31420 raft_consensus.cc:2454] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f783886cc7034f4f9d2ded8c81fd5ac7 in term 1.
I20260808 02:11:49.754647 31095 raft_consensus.cc:2454] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f783886cc7034f4f9d2ded8c81fd5ac7 in term 1.
I20260808 02:11:49.755024 31177 leader_election.cc:302] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d22135282821428eae8471ebea4f910b, f783886cc7034f4f9d2ded8c81fd5ac7; no voters:
I20260808 02:11:49.755270 31516 raft_consensus.cc:2781] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:49.755553 31516 raft_consensus.cc:686] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 1 LEADER]: Becoming Leader. State: Replica: f783886cc7034f4f9d2ded8c81fd5ac7, State: Running, Role: LEADER
I20260808 02:11:49.755897 31516 consensus_queue.cc:237] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } }
I20260808 02:11:49.759402 30914 catalog_manager.cc:5495] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f783886cc7034f4f9d2ded8c81fd5ac7 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f783886cc7034f4f9d2ded8c81fd5ac7" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 38969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 } health_report { overall_health: UNKNOWN } } }
I20260808 02:11:49.778483 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.774610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.5.209.66:38969)
0808 02:11:49.774777 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:49.774783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:49.774800 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:11:49.775011 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:49.775014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:49.775073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:49.775196 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:49.775458 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.775993 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.776884 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:49.777118 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:49.777229 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:49.777609 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:49.778035 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:49.778209 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:49.778298 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:11:49.778335 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:11:49.784722 31095 raft_consensus.cc:1261] T c4b51dabe135457c99b91e194688da01 P d22135282821428eae8471ebea4f910b [term 1 FOLLOWER]: Refusing update from remote peer f783886cc7034f4f9d2ded8c81fd5ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:49.784729 31420 raft_consensus.cc:1261] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 1 FOLLOWER]: Refusing update from remote peer f783886cc7034f4f9d2ded8c81fd5ac7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:49.785486 31520 consensus_queue.cc:1035] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884ae250358244448cc8ab1931316553" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:49.785686 31516 consensus_queue.cc:1035] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22135282821428eae8471ebea4f910b" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:49.793365 31523 mvcc.cc:205] Tried to move back new op lower bound from 7316091329670520832 to 7316091329562988544. Current Snapshot: MvccSnapshot[applied={T|T < 7316091329670520832}]
I20260808 02:11:49.795184 31525 mvcc.cc:205] Tried to move back new op lower bound from 7316091329670520832 to 7316091329562988544. Current Snapshot: MvccSnapshot[applied={T|T < 7316091329670520832}]
I20260808 02:11:49.803514 30913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:49.803701 30913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:49.806128 30913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bdd7e9bf6ea4562b9e270c5a91cc6dd: Sending DeleteTablet for 3 replicas of tablet c4b51dabe135457c99b91e194688da01
I20260808 02:11:49.806993 31235 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b51dabe135457c99b91e194688da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:49 UTC) from {username='slave'} at 127.0.0.1:36282
I20260808 02:11:49.807114 31069 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b51dabe135457c99b91e194688da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:49 UTC) from {username='slave'} at 127.0.0.1:55208
I20260808 02:11:49.807511 31397 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b51dabe135457c99b91e194688da01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:49 UTC) from {username='slave'} at 127.0.0.1:43044
I20260808 02:11:49.807595 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30990
I20260808 02:11:49.807732 31533 tablet_replica.cc:330] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: stopping tablet replica
I20260808 02:11:49.808089 31533 raft_consensus.cc:2229] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:49.808540 31533 raft_consensus.cc:2258] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:49.812306 31533 ts_tablet_manager.cc:1905] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:49.815423 31533 ts_tablet_manager.cc:1918] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:49.815603 31533 log.cc:1196] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/c4b51dabe135457c99b91e194688da01
I20260808 02:11:49.815949 31533 ts_tablet_manager.cc:1939] T c4b51dabe135457c99b91e194688da01 P f783886cc7034f4f9d2ded8c81fd5ac7: Deleting consensus metadata
I20260808 02:11:49.817212 30897 catalog_manager.cc:4841] TS f783886cc7034f4f9d2ded8c81fd5ac7 (127.5.209.66:38969): tablet c4b51dabe135457c99b91e194688da01 (table test-table [id=be12613e6c5b48cb8f24599e721985ed]) successfully deleted
I20260808 02:11:49.817484 31536 tablet_replica.cc:330] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: stopping tablet replica
I20260808 02:11:49.817747 31536 raft_consensus.cc:2229] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:49.818005 31536 raft_consensus.cc:2258] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:49.819484 31536 ts_tablet_manager.cc:1905] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:49.822288 31536 ts_tablet_manager.cc:1918] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:49.822438 31536 log.cc:1196] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/c4b51dabe135457c99b91e194688da01
I20260808 02:11:49.822821 31536 ts_tablet_manager.cc:1939] T c4b51dabe135457c99b91e194688da01 P 884ae250358244448cc8ab1931316553: Deleting consensus metadata
I20260808 02:11:49.824128 30898 catalog_manager.cc:4841] TS 884ae250358244448cc8ab1931316553 (127.5.209.67:33749): tablet c4b51dabe135457c99b91e194688da01 (table test-table [id=be12613e6c5b48cb8f24599e721985ed]) successfully deleted
W20260808 02:11:49.825137 30897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:41087 (error 108)
I20260808 02:11:49.825433 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 31157
I20260808 02:11:49.826413 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:49.826112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:41087 (local address 127.0.0.1:55212)
0808 02:11:49.826273 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:49.826333 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260808 02:11:49.826589 30897 catalog_manager.cc:4584] TS d22135282821428eae8471ebea4f910b (127.5.209.65:41087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4b51dabe135457c99b91e194688da01: Network error: Client connection negotiation failed: client connection to 127.5.209.65:41087: connect: Connection refused (error 111)
I20260808 02:11:49.834141 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 31319
I20260808 02:11:49.850198 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 30877
2026-08-08T02:11:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1093 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:49 dist-test-slave-bw88 krb5kdc[31545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:49 dist-test-slave-bw88 krb5kdc[31545](info): set up 2 sockets
Aug 08 02:11:49 dist-test-slave-bw88 krb5kdc[31545](info): commencing operation
krb5kdc: starting...
W20260808 02:11:51.064659 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:11:51.202293 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:40641
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:51.321430 31562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:51.321740 31562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:51.321818 31562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:51.325860 31562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:51.325937 31562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:51.325974 31562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:51.326031 31562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:51.326066 31562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:51.331272 31562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:40641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:51.332391 31562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:51.333518 31562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:51.342597 31572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:51.342609 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:51.342743 31562 server_base.cc:1034] running on GCE node
W20260808 02:11:51.342618 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:51.343605 31562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:51.344682 31562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:51.345840 31562 hybrid_clock.cc:648] HybridClock initialized: now 1786155111345806 us; error 50 us; skew 500 ppm
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:51.348686 31562 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:11:51.349709 31562 webserver.cc:460] Webserver started at http://127.5.209.126:46373/ using document root <none> and password file <none>
I20260808 02:11:51.350322 31562 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:51.350420 31562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:51.350672 31562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:51.352798 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "826c7800e385415b9883afaaaffd4bf0"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "72062bef2c1bc65e9f96f0fc23bc2904"
server_key_iv: "75ec6c0b9e5a372cf2f2b381fb131889"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.353473 31562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "826c7800e385415b9883afaaaffd4bf0"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "72062bef2c1bc65e9f96f0fc23bc2904"
server_key_iv: "75ec6c0b9e5a372cf2f2b381fb131889"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.357307 31562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 02:11:51.359836 31579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.361120 31562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:51.361291 31562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "826c7800e385415b9883afaaaffd4bf0"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "72062bef2c1bc65e9f96f0fc23bc2904"
server_key_iv: "75ec6c0b9e5a372cf2f2b381fb131889"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.361477 31562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:51.372339 31562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:51.375860 31562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:51.376098 31562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:51.384217 31562 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:40641
I20260808 02:11:51.384246 31652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:40641 every 8 connection(s)
I20260808 02:11:51.385254 31562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:51.388268 31653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:51.389601 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 31562
I20260808 02:11:51.389765 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:51.390195 5957 external_mini_cluster.cc:1428] Setting key 582c01c50631ec74b5bcdad60996032e
I20260808 02:11:51.395067 31653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: Bootstrap starting.
I20260808 02:11:51.397532 31653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:51.398478 31653 log.cc:824] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:51.401274 31653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: No bootstrap required, opened a new log
I20260808 02:11:51.405897 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:51.391863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.5.209.126:40641)
0808 02:11:51.392356 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:51.392372 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:51.392410 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:11:51.393110 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:51.393129 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:51.393560 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:51.393908 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:51.394808 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.395903 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.396815 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.397302 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.397507 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:51.400335 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:51.400345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:51.400363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:51.400398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:51.402607 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.403048 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.403054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.403060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.403208 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.403496 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.403499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.403501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.404109 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:51.404308 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:51.404526 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 02:11:51.404847 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
I20260808 02:11:51.406198 31653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } }
I20260808 02:11:51.406452 31653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:51.406490 31653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826c7800e385415b9883afaaaffd4bf0, State: Initialized, Role: FOLLOWER
I20260808 02:11:51.407012 31653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } }
I20260808 02:11:51.407143 31653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:51.407234 31653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:51.407326 31653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:51.408502 31653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } }
I20260808 02:11:51.408876 31653 leader_election.cc:302] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 826c7800e385415b9883afaaaffd4bf0; no voters:
I20260808 02:11:51.409205 31653 leader_election.cc:288] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:51.409372 31658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:51.409665 31658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [term 1 LEADER]: Becoming Leader. State: Replica: 826c7800e385415b9883afaaaffd4bf0, State: Running, Role: LEADER
I20260808 02:11:51.410025 31658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } }
I20260808 02:11:51.410507 31653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:51.411895 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 826c7800e385415b9883afaaaffd4bf0. Latest consensus state: current_term: 1 leader_uuid: "826c7800e385415b9883afaaaffd4bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } } }
I20260808 02:11:51.411916 31660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "826c7800e385415b9883afaaaffd4bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "826c7800e385415b9883afaaaffd4bf0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 40641 } } }
I20260808 02:11:51.412032 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:51.412048 31660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0 [sys.catalog]: This master's current role is: LEADER
W20260808 02:11:51.415302 31683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:11:51.415373 31683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:11:51.415453 31677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:51.416149 31677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:51.422060 31677 catalog_manager.cc:1347] Generated new cluster ID: 2275cfb17c594e069d4575e7da9c578b
I20260808 02:11:51.422124 31677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:51.444852 31677 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:51.446237 31677 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:51.454707 31677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: Generated new TSK 0
I20260808 02:11:51.455651 31677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.209.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
Entry for principal kudu/127.5.209.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab.
I20260808 02:11:51.497007 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:51.619967 31690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:51.620244 31690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:51.620286 31690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:51.624491 31690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:51.624547 31690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:51.624603 31690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:51.629758 31690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.65.keytab
--principal=kudu/127.5.209.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:51.630743 31690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:51.631729 31690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:51.641816 31695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:51.641816 31696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:51.641856 31699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:51.642134 31690 server_base.cc:1034] running on GCE node
I20260808 02:11:51.642561 31690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:51.643254 31690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:51.644439 31690 hybrid_clock.cc:648] HybridClock initialized: now 1786155111644403 us; error 49 us; skew 500 ppm
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:51.647251 31690 init.cc:377] Logged in from keytab as kudu/127.5.209.65@KRBTEST.COM (short username kudu)
I20260808 02:11:51.648236 31690 webserver.cc:460] Webserver started at http://127.5.209.65:43049/ using document root <none> and password file <none>
I20260808 02:11:51.648835 31690 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:51.648927 31690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:51.649178 31690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:51.651322 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "311da6df29dc496c8895e6d8f29f16e2"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "674e02e5ca5e84c777717b4ff19e2bfe"
server_key_iv: "26f48faea8b6e582b97fc0c0b0add129"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.651913 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "311da6df29dc496c8895e6d8f29f16e2"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "674e02e5ca5e84c777717b4ff19e2bfe"
server_key_iv: "26f48faea8b6e582b97fc0c0b0add129"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.655767 31690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:11:51.658227 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.659439 31690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:11:51.659590 31690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "311da6df29dc496c8895e6d8f29f16e2"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "674e02e5ca5e84c777717b4ff19e2bfe"
server_key_iv: "26f48faea8b6e582b97fc0c0b0add129"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.659699 31690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:51.672708 31690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:51.675834 31690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:51.676043 31690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:51.676630 31690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:51.677567 31690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:51.677635 31690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.677702 31690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:51.677760 31690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.687091 31690 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:44765
I20260808 02:11:51.687124 31859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:44765 every 8 connection(s)
I20260808 02:11:51.687997 31690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:51.695120 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 31690
I20260808 02:11:51.695268 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:51.695647 5957 external_mini_cluster.cc:1428] Setting key 4d6428cfe074aeed5d5b5165dbb401d4
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.65@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:51.702780 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:51.689984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:41989 (local address 127.5.209.126:40641)
0808 02:11:51.690150 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:51.690155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:51.691079 (+ 924us) server_negotiation.cc:407] Connection header received
0808 02:11:51.691949 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:51.691952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:51.692001 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:51.692081 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:51.693272 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.693884 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.695052 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.695412 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.695532 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:51.698329 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:51.698336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:51.698342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:51.698378 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:51.700087 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.700849 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.700852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.700853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.700931 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.701440 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.701447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.701451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.701719 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:51.701834 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:51.702373 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 02:11:51.702516 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:11:51.703562 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:51.690407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:40641 (local address 127.5.209.65:41989)
0808 02:11:51.690899 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:51.690955 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:51.691746 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:51.692252 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:51.692259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:51.692640 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:51.693135 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:51.693147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.694006 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.694009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:51.694905 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:51.694915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.695571 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.695574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:51.695761 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:51.696538 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:51.696563 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:51.698159 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:51.700230 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:51.700238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:51.700258 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:51.700729 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:51.701056 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:51.701061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:51.701065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:51.701290 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:51.701838 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:51.701844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:51.702097 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 02:11:51.702272 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:11:51.702572 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260808 02:11:51.705029 31860 heartbeater.cc:347] Connected to a master server at 127.5.209.126:40641
I20260808 02:11:51.705286 31860 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:51.705826 31860 heartbeater.cc:511] Master 127.5.209.126:40641 requested a full tablet report, sending...
I20260808 02:11:51.707477 31606 ts_manager.cc:194] Registered new tserver with Master: 311da6df29dc496c8895e6d8f29f16e2 (127.5.209.65:44765)
WARNING: no policy specified for kudu/127.5.209.66@KRBTEST.COM; defaulting to no policy
I20260808 02:11:51.708701 31606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.65@KRBTEST.COM'} at 127.5.209.65:41989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
Entry for principal kudu/127.5.209.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab.
I20260808 02:11:51.726497 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:51.844445 31867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:51.844730 31867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:51.844810 31867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:51.848791 31867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:51.848883 31867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:51.848974 31867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:51.854128 31867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.66.keytab
--principal=kudu/127.5.209.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:51.855188 31867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:51.856228 31867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:51.866741 31874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:51.866750 31876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:51.866762 31872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:51.867020 31867 server_base.cc:1034] running on GCE node
I20260808 02:11:51.867404 31867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:51.867977 31867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:51.869140 31867 hybrid_clock.cc:648] HybridClock initialized: now 1786155111869105 us; error 46 us; skew 500 ppm
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:51.871989 31867 init.cc:377] Logged in from keytab as kudu/127.5.209.66@KRBTEST.COM (short username kudu)
I20260808 02:11:51.872965 31867 webserver.cc:460] Webserver started at http://127.5.209.66:44425/ using document root <none> and password file <none>
I20260808 02:11:51.873649 31867 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:51.873754 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:51.874002 31867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:51.876070 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "e87d0685188b4ec390b4ef234d8ed98e"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "c7d464db13af4652c2041b57744450b1"
server_key_iv: "2f974824175865cd701f1fca27c81a54"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.876674 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "e87d0685188b4ec390b4ef234d8ed98e"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "c7d464db13af4652c2041b57744450b1"
server_key_iv: "2f974824175865cd701f1fca27c81a54"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.880430 31867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 02:11:51.883097 31887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.884371 31867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:11:51.884547 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "e87d0685188b4ec390b4ef234d8ed98e"
format_stamp: "Formatted at 2026-08-08 02:11:51 on dist-test-slave-bw88"
server_key: "c7d464db13af4652c2041b57744450b1"
server_key_iv: "2f974824175865cd701f1fca27c81a54"
server_key_version: "encryptionkey@0"
I20260808 02:11:51.884676 31867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:51.904641 31867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:51.907684 31867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:51.907904 31867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:51.908555 31867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:51.909513 31867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:51.909584 31867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.909658 31867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:51.909703 31867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:51.919324 31867 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:39205
I20260808 02:11:51.919364 32029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:39205 every 8 connection(s)
I20260808 02:11:51.920289 31867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:51.923277 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 31867
I20260808 02:11:51.923411 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:51.923763 5957 external_mini_cluster.cc:1428] Setting key edfe4ef139856c78e82e317d5e6e7a9b
Aug 08 02:11:51 dist-test-slave-bw88 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155111, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.66@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.209.67@KRBTEST.COM; defaulting to no policy
I20260808 02:11:51.935086 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:51.922265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:35897 (local address 127.5.209.126:40641)
0808 02:11:51.922407 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:51.922411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:51.923388 (+ 977us) server_negotiation.cc:407] Connection header received
0808 02:11:51.924622 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:51.924625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:51.924682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:51.924773 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:51.926405 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.926980 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.928427 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.928660 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.928768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:51.931298 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:51.931302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:51.931305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:51.931339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:51.932845 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.933399 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.933403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.933405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.933491 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:51.933845 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:51.933849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:51.933850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:51.934049 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:51.934160 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:51.934769 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 02:11:51.934899 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:11:51.935910 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:51.922660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:40641 (local address 127.5.209.66:35897)
0808 02:11:51.923182 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:51.923248 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:51.924338 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:51.924937 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:51.924944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:51.925549 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:51.926248 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:51.926272 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.927121 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.927128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:51.928285 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:51.928296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:51.928822 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:51.928827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:51.928998 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:51.929663 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:51.929681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:51.931123 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:51.932967 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:51.932973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:51.932988 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:51.933251 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:51.933587 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:51.933590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:51.933592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:51.933735 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:51.934176 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:51.934185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:51.934431 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 02:11:51.934682 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 02:11:51.935029 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260808 02:11:51.937182 32030 heartbeater.cc:347] Connected to a master server at 127.5.209.126:40641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.67@KRBTEST.COM" created.
I20260808 02:11:51.937557 32030 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:51.938086 32030 heartbeater.cc:511] Master 127.5.209.126:40641 requested a full tablet report, sending...
I20260808 02:11:51.939239 31606 ts_manager.cc:194] Registered new tserver with Master: e87d0685188b4ec390b4ef234d8ed98e (127.5.209.66:39205)
I20260808 02:11:51.940116 31606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.66@KRBTEST.COM'} at 127.5.209.66:35897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
Entry for principal kudu/127.5.209.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab.
I20260808 02:11:51.954481 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:52.072588 32036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:52.072835 32036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:52.072877 32036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:52.076831 32036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:52.076891 32036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:52.076948 32036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:52.082366 32036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.67.keytab
--principal=kudu/127.5.209.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:40641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:52.083421 32036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:52.084434 32036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:52.094503 32044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.094586 32042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.094497 32041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:52.094933 32036 server_base.cc:1034] running on GCE node
I20260808 02:11:52.095345 32036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:52.096001 32036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:52.097205 32036 hybrid_clock.cc:648] HybridClock initialized: now 1786155112097190 us; error 26 us; skew 500 ppm
Aug 08 02:11:52 dist-test-slave-bw88 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155112, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:52.100054 32036 init.cc:377] Logged in from keytab as kudu/127.5.209.67@KRBTEST.COM (short username kudu)
I20260808 02:11:52.100883 32036 webserver.cc:460] Webserver started at http://127.5.209.67:42295/ using document root <none> and password file <none>
I20260808 02:11:52.101500 32036 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:52.101568 32036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:52.101742 32036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:52.103750 32036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "a6982e117a294bdaa574664e133a7640"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "25a0988a43b810f268cadfaa2995711b"
server_key_iv: "38b2a2deb400691c0b3cc44412a6b1df"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.104281 32036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "a6982e117a294bdaa574664e133a7640"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "25a0988a43b810f268cadfaa2995711b"
server_key_iv: "38b2a2deb400691c0b3cc44412a6b1df"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.108033 32036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 02:11:52.110606 32052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.112020 32036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:11:52.112164 32036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "a6982e117a294bdaa574664e133a7640"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "25a0988a43b810f268cadfaa2995711b"
server_key_iv: "38b2a2deb400691c0b3cc44412a6b1df"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.112257 32036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:52.138804 32036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:52.142266 32036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:52.142424 32036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:52.142997 32036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:52.143821 32036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:52.143869 32036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.143911 32036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:52.143932 32036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.153381 32036 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:35793
I20260808 02:11:52.153420 32192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:35793 every 8 connection(s)
I20260808 02:11:52.154317 32036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:52.162236 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 32036
I20260808 02:11:52.162415 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:52.162772 5957 external_mini_cluster.cc:1428] Setting key 0f8ab2a069923ad842e0f58003bf5b31
Aug 08 02:11:52 dist-test-slave-bw88 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155112, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.67@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:11:52.168642 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.156260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:51115 (local address 127.5.209.126:40641)
0808 02:11:52.156405 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.156408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.157283 (+ 875us) server_negotiation.cc:407] Connection header received
0808 02:11:52.158280 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.158283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.158333 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.158408 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:52.159516 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.160048 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.161204 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.161551 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.161644 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.164581 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:52.164586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:52.164588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:52.164613 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:52.166646 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:52.167182 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:52.167185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:52.167186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:52.167276 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:52.167593 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:52.167596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:52.167597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:52.167782 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:52.167863 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.168415 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.168520 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:11:52.169579 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.156606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:40641 (local address 127.5.209.67:51115)
0808 02:11:52.157098 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.157140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.158068 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.158537 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.158543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.158898 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:52.159385 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.159399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.160158 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.160161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.161047 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.161058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.161708 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.161711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.161858 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.162481 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:11:52.162517 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:52.164376 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:52.166782 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:52.166787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:52.166799 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:52.167078 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:52.167363 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:11:52.167365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:11:52.167367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:11:52.167507 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:11:52.167911 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:52.167916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:52.168172 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.168354 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.168641 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":130,"threads_started":1}
I20260808 02:11:52.170763 32194 heartbeater.cc:347] Connected to a master server at 127.5.209.126:40641
I20260808 02:11:52.171041 32194 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:52.171626 32194 heartbeater.cc:511] Master 127.5.209.126:40641 requested a full tablet report, sending...
I20260808 02:11:52.172703 31606 ts_manager.cc:194] Registered new tserver with Master: a6982e117a294bdaa574664e133a7640 (127.5.209.67:35793)
I20260808 02:11:52.173234 31606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.209.67@KRBTEST.COM'} at 127.5.209.67:51115
I20260808 02:11:52.177251 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:52.187088 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.179438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.5.209.126:40641)
0808 02:11:52.179603 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.179608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.179754 (+ 146us) server_negotiation.cc:407] Connection header received
0808 02:11:52.179923 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.179929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.179974 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.180057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:52.180806 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.181379 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.182259 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.182489 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.182600 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.183949 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:52.183953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:52.183956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:52.183981 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:52.185281 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:52.185799 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:52.185803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:52.185805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:52.185890 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:52.186186 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:52.186190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:52.186192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:52.186414 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:52.186575 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.186810 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.186926 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:11:52.190078 31606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:52.193086 31606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:52.208256 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.202207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44765 (local address 127.0.0.1:59706)
0808 02:11:52.203369 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.203381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.203459 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.204167 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.204171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.204189 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:52.204270 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.204275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205622 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.205625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.206804 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.206816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.207708 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.207712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.207816 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.207855 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.207973 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.208105 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":89,"thread_start_us":116,"threads_started":1}
I20260808 02:11:52.208271 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.202768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.5.209.65:44765)
0808 02:11:52.203058 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.203077 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.203568 (+ 491us) server_negotiation.cc:407] Connection header received
0808 02:11:52.203653 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.203659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.203829 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.203976 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:52.204394 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205499 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.206935 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.207570 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.207695 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.207856 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.208013 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.208112 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260808 02:11:52.208585 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.202616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.5.209.66:39205)
0808 02:11:52.203288 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.203299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.203324 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:11:52.203407 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.203412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.203607 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.203794 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:52.204243 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205446 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.207058 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.207827 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.207919 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.208035 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.208349 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.208441 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":126,"threads_started":1}
I20260808 02:11:52.208498 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.201851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:35793 (local address 127.0.0.1:43534)
0808 02:11:52.202772 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.202825 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.202966 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.203788 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.203794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.203836 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:52.204048 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.204057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205167 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.205172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.206689 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.206701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.208042 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.208046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.208140 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.208194 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.208268 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.208357 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":92,"spinlock_wait_cycles":4608,"thread_start_us":97,"threads_started":1}
I20260808 02:11:52.208909 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.201959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:39205 (local address 127.0.0.1:48040)
0808 02:11:52.203107 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.203122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.203197 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.203788 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.203794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.203833 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:52.204090 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.204100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205602 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.205605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.206930 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.206938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.208098 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.208102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.208195 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.208214 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.208277 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.208784 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":630,"spinlock_wait_cycles":5248,"thread_start_us":116,"threads_started":1}
I20260808 02:11:52.209373 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.202186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.5.209.67:35793)
0808 02:11:52.202923 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.202936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.202961 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:11:52.203197 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.203205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.203441 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.203602 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:52.204191 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.205030 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.206823 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.207699 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.207812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.207974 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.208615 (+ 641us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.208725 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":138,"threads_started":1}
I20260808 02:11:52.211971 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 93fdc26b093b44c7a7bd0b584e6436ee (DEFAULT_TABLE table=test-table [id=03c291f5989b4dd89d96c229e8d7371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:52.211982 31773 tablet_service.cc:1467] Processing CreateTablet for tablet 93fdc26b093b44c7a7bd0b584e6436ee (DEFAULT_TABLE table=test-table [id=03c291f5989b4dd89d96c229e8d7371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:52.211982 31948 tablet_service.cc:1467] Processing CreateTablet for tablet 93fdc26b093b44c7a7bd0b584e6436ee (DEFAULT_TABLE table=test-table [id=03c291f5989b4dd89d96c229e8d7371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:52.213032 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fdc26b093b44c7a7bd0b584e6436ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:52.213050 31773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fdc26b093b44c7a7bd0b584e6436ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:52.213050 31948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fdc26b093b44c7a7bd0b584e6436ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:52.219834 32214 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Bootstrap starting.
I20260808 02:11:52.221042 32215 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: Bootstrap starting.
I20260808 02:11:52.221675 32218 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Bootstrap starting.
I20260808 02:11:52.223001 32214 tablet_bootstrap.cc:654] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:52.223294 32215 tablet_bootstrap.cc:654] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:52.224320 32218 tablet_bootstrap.cc:654] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:52.224284 32215 log.cc:824] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:52.224654 32214 log.cc:824] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:52.225148 32218 log.cc:824] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:52.226678 32215 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: No bootstrap required, opened a new log
I20260808 02:11:52.226886 32215 ts_tablet_manager.cc:1397] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 02:11:52.227331 32218 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: No bootstrap required, opened a new log
I20260808 02:11:52.227515 32218 ts_tablet_manager.cc:1397] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 02:11:52.228196 32214 tablet_bootstrap.cc:492] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: No bootstrap required, opened a new log
I20260808 02:11:52.228387 32214 ts_tablet_manager.cc:1397] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260808 02:11:52.231575 32214 raft_consensus.cc:348] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.231635 32215 raft_consensus.cc:348] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.231635 32218 raft_consensus.cc:348] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.231851 32214 raft_consensus.cc:374] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:52.231889 32218 raft_consensus.cc:374] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:52.231889 32215 raft_consensus.cc:374] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:52.231901 32214 raft_consensus.cc:729] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6982e117a294bdaa574664e133a7640, State: Initialized, Role: FOLLOWER
I20260808 02:11:52.231940 32215 raft_consensus.cc:729] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311da6df29dc496c8895e6d8f29f16e2, State: Initialized, Role: FOLLOWER
I20260808 02:11:52.231950 32218 raft_consensus.cc:729] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87d0685188b4ec390b4ef234d8ed98e, State: Initialized, Role: FOLLOWER
I20260808 02:11:52.232353 32214 consensus_queue.cc:260] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.232434 32218 consensus_queue.cc:260] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.232434 32215 consensus_queue.cc:260] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.233027 32194 heartbeater.cc:503] Master 127.5.209.126:40641 was elected leader, sending a full tablet report...
I20260808 02:11:52.233433 32214 ts_tablet_manager.cc:1428] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 02:11:52.233436 32218 ts_tablet_manager.cc:1428] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 02:11:52.233436 32215 ts_tablet_manager.cc:1428] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 02:11:52.233772 31860 heartbeater.cc:503] Master 127.5.209.126:40641 was elected leader, sending a full tablet report...
I20260808 02:11:52.233772 32030 heartbeater.cc:503] Master 127.5.209.126:40641 was elected leader, sending a full tablet report...
I20260808 02:11:52.362304 32224 raft_consensus.cc:482] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:52.362605 32224 raft_consensus.cc:504] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.364084 32224 leader_election.cc:288] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6982e117a294bdaa574664e133a7640 (127.5.209.67:35793), 311da6df29dc496c8895e6d8f29f16e2 (127.5.209.65:44765)
I20260808 02:11:52.368229 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44765 (local address 127.5.209.66:42863)
0808 02:11:52.364725 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.364740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.364849 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.365204 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.365207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.365227 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:52.365418 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.365431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.366260 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.366265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.367072 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.367081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.367772 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.367776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.367883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.367907 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.368013 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.368089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1}
I20260808 02:11:52.368229 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.364640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:42863 (local address 127.5.209.65:44765)
0808 02:11:52.364794 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.364799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.364815 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:11:52.365000 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.365004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.365051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.365145 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:52.365585 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.366145 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.367208 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.367639 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.367773 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.367831 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.368023 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.368100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:11:52.368464 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.364588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:41905 (local address 127.5.209.67:35793)
0808 02:11:52.364740 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.364745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.365119 (+ 374us) server_negotiation.cc:407] Connection header received
0808 02:11:52.365314 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.365317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.365424 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.365503 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:52.365789 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.366410 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.367478 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.367890 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.368026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.368082 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.368283 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.368354 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:11:52.368484 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.364486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:35793 (local address 127.5.209.66:41905)
0808 02:11:52.364983 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.364995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.365178 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.365515 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.365518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.365536 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:52.365673 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.365679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.366525 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.366531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.367341 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.367350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.368027 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.368031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.368153 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.368172 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.368264 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.368335 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":43,"thread_start_us":139,"threads_started":1}
I20260808 02:11:52.368979 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fdc26b093b44c7a7bd0b584e6436ee" candidate_uuid: "e87d0685188b4ec390b4ef234d8ed98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311da6df29dc496c8895e6d8f29f16e2" is_pre_election: true
I20260808 02:11:52.369102 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fdc26b093b44c7a7bd0b584e6436ee" candidate_uuid: "e87d0685188b4ec390b4ef234d8ed98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6982e117a294bdaa574664e133a7640" is_pre_election: true
I20260808 02:11:52.369385 31801 raft_consensus.cc:2454] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87d0685188b4ec390b4ef234d8ed98e in term 0.
I20260808 02:11:52.369459 32138 raft_consensus.cc:2454] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87d0685188b4ec390b4ef234d8ed98e in term 0.
I20260808 02:11:52.369972 31890 leader_election.cc:302] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311da6df29dc496c8895e6d8f29f16e2, e87d0685188b4ec390b4ef234d8ed98e; no voters:
I20260808 02:11:52.370296 32224 raft_consensus.cc:2781] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:52.370407 32224 raft_consensus.cc:482] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:52.370438 32224 raft_consensus.cc:3037] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:52.371512 32224 raft_consensus.cc:504] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.371879 32224 leader_election.cc:288] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [CANDIDATE]: Term 1 election: Requested vote from peers a6982e117a294bdaa574664e133a7640 (127.5.209.67:35793), 311da6df29dc496c8895e6d8f29f16e2 (127.5.209.65:44765)
I20260808 02:11:52.372294 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fdc26b093b44c7a7bd0b584e6436ee" candidate_uuid: "e87d0685188b4ec390b4ef234d8ed98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6982e117a294bdaa574664e133a7640"
I20260808 02:11:52.372294 31801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fdc26b093b44c7a7bd0b584e6436ee" candidate_uuid: "e87d0685188b4ec390b4ef234d8ed98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311da6df29dc496c8895e6d8f29f16e2"
I20260808 02:11:52.372432 32138 raft_consensus.cc:3037] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:52.372432 31801 raft_consensus.cc:3037] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:52.373817 32138 raft_consensus.cc:2454] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87d0685188b4ec390b4ef234d8ed98e in term 1.
I20260808 02:11:52.373817 31801 raft_consensus.cc:2454] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87d0685188b4ec390b4ef234d8ed98e in term 1.
I20260808 02:11:52.374193 31891 leader_election.cc:302] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 311da6df29dc496c8895e6d8f29f16e2, a6982e117a294bdaa574664e133a7640, e87d0685188b4ec390b4ef234d8ed98e; no voters:
I20260808 02:11:52.374437 32224 raft_consensus.cc:2781] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:52.374763 32224 raft_consensus.cc:686] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 1 LEADER]: Becoming Leader. State: Replica: e87d0685188b4ec390b4ef234d8ed98e, State: Running, Role: LEADER
I20260808 02:11:52.375156 32224 consensus_queue.cc:237] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } }
I20260808 02:11:52.378645 31605 catalog_manager.cc:5495] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e reported cstate change: term changed from 0 to 1, leader changed from <none> to e87d0685188b4ec390b4ef234d8ed98e (127.5.209.66). New cstate: current_term: 1 leader_uuid: "e87d0685188b4ec390b4ef234d8ed98e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e87d0685188b4ec390b4ef234d8ed98e" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 } health_report { overall_health: UNKNOWN } } }
I20260808 02:11:52.398200 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.394292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.5.209.66:39205)
0808 02:11:52.394414 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.394418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.394433 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:11:52.394574 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.394578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.394645 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.394755 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:52.394975 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.395794 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.396617 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.396921 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.397046 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.397288 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:52.397792 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:52.397889 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.398019 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.398075 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 02:11:52.404251 32138 raft_consensus.cc:1261] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 1 FOLLOWER]: Refusing update from remote peer e87d0685188b4ec390b4ef234d8ed98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:52.404251 31801 raft_consensus.cc:1261] T 93fdc26b093b44c7a7bd0b584e6436ee P 311da6df29dc496c8895e6d8f29f16e2 [term 1 FOLLOWER]: Refusing update from remote peer e87d0685188b4ec390b4ef234d8ed98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:52.405021 32227 consensus_queue.cc:1035] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6982e117a294bdaa574664e133a7640" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:52.405224 32224 consensus_queue.cc:1035] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "311da6df29dc496c8895e6d8f29f16e2" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 44765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:11:52.405431 32195 tablet.cc:2369] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:52.423879 31605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:52.424099 31605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:52.426743 31605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 826c7800e385415b9883afaaaffd4bf0: Sending DeleteTablet for 3 replicas of tablet 93fdc26b093b44c7a7bd0b584e6436ee
I20260808 02:11:52.427529 32113 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fdc26b093b44c7a7bd0b584e6436ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:52 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:43534
I20260808 02:11:52.427613 31948 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fdc26b093b44c7a7bd0b584e6436ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:52 UTC) from {username='kudu', principal='kudu/127.5.209.126@KRBTEST.COM'} at 127.0.0.1:48040
I20260808 02:11:52.428164 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 31690
I20260808 02:11:52.434767 32245 tablet_replica.cc:330] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: stopping tablet replica
I20260808 02:11:52.435271 32245 raft_consensus.cc:2229] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:52.435976 32245 raft_consensus.cc:2258] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:52.438496 32245 ts_tablet_manager.cc:1905] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:52.442932 32245 ts_tablet_manager.cc:1918] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:52.443073 32245 log.cc:1196] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/93fdc26b093b44c7a7bd0b584e6436ee
I20260808 02:11:52.443441 32245 ts_tablet_manager.cc:1939] T 93fdc26b093b44c7a7bd0b584e6436ee P e87d0685188b4ec390b4ef234d8ed98e: Deleting consensus metadata
I20260808 02:11:52.445298 32244 tablet_replica.cc:330] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: stopping tablet replica
I20260808 02:11:52.445850 32244 raft_consensus.cc:2229] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:52.446314 32244 raft_consensus.cc:2258] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:52.448370 32244 ts_tablet_manager.cc:1905] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:52.452669 32244 ts_tablet_manager.cc:1918] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:52.452817 32244 log.cc:1196] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/93fdc26b093b44c7a7bd0b584e6436ee
I20260808 02:11:52.453280 32244 ts_tablet_manager.cc:1939] T 93fdc26b093b44c7a7bd0b584e6436ee P a6982e117a294bdaa574664e133a7640: Deleting consensus metadata
I20260808 02:11:52.455210 31584 catalog_manager.cc:4841] TS a6982e117a294bdaa574664e133a7640 (127.5.209.67:35793): tablet 93fdc26b093b44c7a7bd0b584e6436ee (table test-table [id=03c291f5989b4dd89d96c229e8d7371f]) successfully deleted
I20260808 02:11:52.458832 31581 catalog_manager.cc:4841] TS e87d0685188b4ec390b4ef234d8ed98e (127.5.209.66:39205): tablet 93fdc26b093b44c7a7bd0b584e6436ee (table test-table [id=03c291f5989b4dd89d96c229e8d7371f]) successfully deleted
W20260808 02:11:52.465864 31583 connection.cc:535] client connection to 127.5.209.65:44765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 02:11:52.466054 31583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:11:52.466168 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 31867
I20260808 02:11:52.469728 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.468722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:44765 (local address 127.0.0.1:59990)
0808 02:11:52.469521 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.469613 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:44765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665}
W20260808 02:11:52.469934 31583 catalog_manager.cc:4584] TS 311da6df29dc496c8895e6d8f29f16e2 (127.5.209.65:44765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93fdc26b093b44c7a7bd0b584e6436ee: Network error: Client connection negotiation failed: client connection to 127.5.209.65:44765: connect: Connection refused (error 111)
I20260808 02:11:52.476053 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 32036
I20260808 02:11:52.486958 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 31562
2026-08-08T02:11:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T02:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:52Z Disabled control of system clock
I20260808 02:11:52.522388 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:37401
--webserver_interface=127.5.209.126
--webserver_port=0
--builtin_ntp_servers=127.5.209.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:37401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:52.638597 32251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:52.638867 32251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:52.638902 32251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:52.642918 32251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:52.642982 32251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:52.642997 32251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:52.643014 32251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:52.643029 32251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:52.647923 32251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:37401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:37401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:52.649077 32251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:52.650578 32251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:52.659335 32256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.659370 32257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.659379 32259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:52.659689 32251 server_base.cc:1034] running on GCE node
I20260808 02:11:52.660223 32251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:52.661396 32251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:52.662608 32251 hybrid_clock.cc:648] HybridClock initialized: now 1786155112662591 us; error 35 us; skew 500 ppm
I20260808 02:11:52.668951 32251 webserver.cc:460] Webserver started at http://127.5.209.126:46877/ using document root <none> and password file <none>
I20260808 02:11:52.669649 32251 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:52.669746 32251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:52.669996 32251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:52.672086 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "be87051ae95148139d015214d7e473e1"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "b290b9cfef1fd90298448bdb2f69b837"
server_key_iv: "1094f96b34539c4a3da4bb6d365dbe96"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.672675 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "be87051ae95148139d015214d7e473e1"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "b290b9cfef1fd90298448bdb2f69b837"
server_key_iv: "1094f96b34539c4a3da4bb6d365dbe96"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.677613 32251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:11:52.681195 32266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.682828 32251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:11:52.682971 32251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "be87051ae95148139d015214d7e473e1"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "b290b9cfef1fd90298448bdb2f69b837"
server_key_iv: "1094f96b34539c4a3da4bb6d365dbe96"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.683110 32251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:52.694983 32251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:52.695653 32251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:52.695806 32251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:52.703187 32251 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:37401
I20260808 02:11:52.703194 32334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:37401 every 8 connection(s)
I20260808 02:11:52.704177 32251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:52.707424 32335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:52.708730 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 32251
I20260808 02:11:52.708889 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:52.709210 5957 external_mini_cluster.cc:1428] Setting key 98ba93e5c535f328b26ea1f10543921d
I20260808 02:11:52.713917 32335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: Bootstrap starting.
I20260808 02:11:52.716631 32335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:52.717507 32335 log.cc:824] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:52.719026 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.710807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.5.209.126:37401)
0808 02:11:52.711424 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.711440 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.711482 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:11:52.712119 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.712136 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.712522 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.712794 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:52.713178 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.714312 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.715507 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.715960 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.716075 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.716991 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:52.716996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:52.717009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:52.717441 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:52.717532 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:52.717557 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:52.717718 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.717877 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.718167 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260808 02:11:52.719759 32335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: No bootstrap required, opened a new log
I20260808 02:11:52.723740 32335 raft_consensus.cc:348] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } }
I20260808 02:11:52.723979 32335 raft_consensus.cc:374] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:52.724066 32335 raft_consensus.cc:729] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be87051ae95148139d015214d7e473e1, State: Initialized, Role: FOLLOWER
I20260808 02:11:52.724479 32335 consensus_queue.cc:260] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } }
I20260808 02:11:52.724627 32335 raft_consensus.cc:388] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:52.724712 32335 raft_consensus.cc:482] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:52.724850 32335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:52.725841 32335 raft_consensus.cc:504] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } }
I20260808 02:11:52.726191 32335 leader_election.cc:302] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be87051ae95148139d015214d7e473e1; no voters:
I20260808 02:11:52.726517 32335 leader_election.cc:288] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:52.726626 32341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:52.726882 32341 raft_consensus.cc:686] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [term 1 LEADER]: Becoming Leader. State: Replica: be87051ae95148139d015214d7e473e1, State: Running, Role: LEADER
I20260808 02:11:52.727231 32341 consensus_queue.cc:237] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } }
I20260808 02:11:52.727819 32335 sys_catalog.cc:564] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:52.729063 32343 sys_catalog.cc:455] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be87051ae95148139d015214d7e473e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } } }
I20260808 02:11:52.729234 32343 sys_catalog.cc:458] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:52.729069 32345 sys_catalog.cc:455] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be87051ae95148139d015214d7e473e1. Latest consensus state: current_term: 1 leader_uuid: "be87051ae95148139d015214d7e473e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be87051ae95148139d015214d7e473e1" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 37401 } } }
I20260808 02:11:52.729575 32345 sys_catalog.cc:458] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1 [sys.catalog]: This master's current role is: LEADER
W20260808 02:11:52.732805 32361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:11:52.732874 32361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:11:52.732954 32354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:52.733781 32354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:52.739185 32354 catalog_manager.cc:1347] Generated new cluster ID: 18081dbe40564abab59c6d3148fca8f8
I20260808 02:11:52.739252 32354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:52.757102 32354 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:52.758121 32354 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:52.762140 32354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: Generated new TSK 0
I20260808 02:11:52.762847 32354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:52.776317 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.65:0
--local_ip_for_outbound_sockets=127.5.209.65
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--builtin_ntp_servers=127.5.209.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:11:52.898293 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:52.898609 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:52.898651 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:52.902752 32365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:52.902809 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:52.902863 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.65
I20260808 02:11:52.907792 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.209.65
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.65
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:52.908951 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:52.910457 32365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:52.920461 32375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.920470 32372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:52.920506 32373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:52.920720 32365 server_base.cc:1034] running on GCE node
I20260808 02:11:52.921216 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:52.921945 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:52.923141 32365 hybrid_clock.cc:648] HybridClock initialized: now 1786155112923119 us; error 31 us; skew 500 ppm
I20260808 02:11:52.925081 32365 webserver.cc:460] Webserver started at http://127.5.209.65:46701/ using document root <none> and password file <none>
I20260808 02:11:52.925745 32365 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:52.925846 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:52.926092 32365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:52.928264 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data/instance:
uuid: "08fe3aa92016405cb694510c0aa10c31"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "d9c9e3494f81cd135ec1e65fdf5f1319"
server_key_iv: "6a96713b88a4f0dd45477587cd2ba7e0"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.928937 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance:
uuid: "08fe3aa92016405cb694510c0aa10c31"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "d9c9e3494f81cd135ec1e65fdf5f1319"
server_key_iv: "6a96713b88a4f0dd45477587cd2ba7e0"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.932911 32365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:11:52.935925 32381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.937326 32365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:11:52.937589 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal
uuid: "08fe3aa92016405cb694510c0aa10c31"
format_stamp: "Formatted at 2026-08-08 02:11:52 on dist-test-slave-bw88"
server_key: "d9c9e3494f81cd135ec1e65fdf5f1319"
server_key_iv: "6a96713b88a4f0dd45477587cd2ba7e0"
server_key_version: "encryptionkey@0"
I20260808 02:11:52.937721 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:52.957748 32365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:52.958498 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:52.958710 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:52.959343 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:52.960345 32365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:52.960420 32365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.960507 32365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:52.960563 32365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:52.970192 32365 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.65:34283
I20260808 02:11:52.970229 32533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.65:34283 every 8 connection(s)
I20260808 02:11:52.971132 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/data/info.pb
I20260808 02:11:52.975106 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 32365
I20260808 02:11:52.975241 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-0/wal/instance
I20260808 02:11:52.975565 5957 external_mini_cluster.cc:1428] Setting key f3e3c96365abe73974ebcc75f5753933
I20260808 02:11:52.977989 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.66:0
--local_ip_for_outbound_sockets=127.5.209.66
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--builtin_ntp_servers=127.5.209.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:52.982439 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.973050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.65:58519 (local address 127.5.209.126:37401)
0808 02:11:52.973210 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:52.973215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:52.974572 (+ 1357us) server_negotiation.cc:407] Connection header received
0808 02:11:52.976577 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:52.976581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:52.976638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:52.976730 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:52.977909 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.978698 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.980162 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.980379 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.980467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.981888 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:52.981892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:52.981894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:52.981916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:52.981936 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:52.981945 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:52.982041 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:52.982223 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:11:52.982292 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:11:52.983670 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:52.973559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:37401 (local address 127.5.209.65:58519)
0808 02:11:52.974340 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:52.974405 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:52.976329 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:52.976943 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:52.976952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:52.977066 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:52.977752 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.977767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.978852 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.978855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.980020 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:52.980037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:52.980558 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:52.980564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:52.980809 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:52.981578 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:52.981580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:52.981654 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:52.981727 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:52.982044 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:52.982057 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:52.982066 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:11:52.982206 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:11:52.982529 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260808 02:11:52.985087 32534 heartbeater.cc:347] Connected to a master server at 127.5.209.126:37401
I20260808 02:11:52.985419 32534 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:52.986011 32534 heartbeater.cc:511] Master 127.5.209.126:37401 requested a full tablet report, sending...
I20260808 02:11:52.987934 32288 ts_manager.cc:194] Registered new tserver with Master: 08fe3aa92016405cb694510c0aa10c31 (127.5.209.65:34283)
I20260808 02:11:52.989768 32288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.65:58519
W20260808 02:11:53.101199 32537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:53.101542 32537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:53.101589 32537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:53.105546 32537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:53.105592 32537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:53.105646 32537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.66
I20260808 02:11:53.110319 32537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.209.66
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.66
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:53.111348 32537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:53.112715 32537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:53.122623 32542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:53.122622 32543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:53.122674 32545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:53.122941 32537 server_base.cc:1034] running on GCE node
I20260808 02:11:53.123322 32537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:53.124056 32537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:53.125252 32537 hybrid_clock.cc:648] HybridClock initialized: now 1786155113125226 us; error 46 us; skew 500 ppm
I20260808 02:11:53.127074 32537 webserver.cc:460] Webserver started at http://127.5.209.66:35087/ using document root <none> and password file <none>
I20260808 02:11:53.127735 32537 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:53.127843 32537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:53.128062 32537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:53.130321 32537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data/instance:
uuid: "e32d5bb786714e4d998a6efb9ff6b075"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "17f7fd24253129f0ddebdc1bcdf85281"
server_key_iv: "814850de2a8e03f5f94fef1e63c75d66"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.130899 32537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance:
uuid: "e32d5bb786714e4d998a6efb9ff6b075"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "17f7fd24253129f0ddebdc1bcdf85281"
server_key_iv: "814850de2a8e03f5f94fef1e63c75d66"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.134609 32537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:11:53.137032 32551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.138262 32537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:53.138402 32537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal
uuid: "e32d5bb786714e4d998a6efb9ff6b075"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "17f7fd24253129f0ddebdc1bcdf85281"
server_key_iv: "814850de2a8e03f5f94fef1e63c75d66"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.138540 32537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:53.162425 32537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:53.163206 32537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:53.163427 32537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:53.164074 32537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:53.165091 32537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:53.165166 32537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.165243 32537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:53.165287 32537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.174721 32537 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.66:44663
I20260808 02:11:53.174746 32689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.66:44663 every 8 connection(s)
I20260808 02:11:53.175712 32537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/data/info.pb
I20260808 02:11:53.176543 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 32537
I20260808 02:11:53.176685 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal/instance
I20260808 02:11:53.177107 5957 external_mini_cluster.cc:1428] Setting key 3dddd70e0f1b03daf7c1f631e7d278ab
I20260808 02:11:53.179468 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.209.67:0
--local_ip_for_outbound_sockets=127.5.209.67
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--builtin_ntp_servers=127.5.209.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:11:53.186836 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.178068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:35905 (local address 127.5.209.126:37401)
0808 02:11:53.178246 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.178252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.179423 (+ 1171us) server_negotiation.cc:407] Connection header received
0808 02:11:53.181689 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.181696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.181765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.181864 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:53.182763 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.183434 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.184634 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.184857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.184975 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.186276 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:53.186280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:53.186282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:53.186294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:53.186331 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:53.186338 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:53.186417 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.186577 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.186641 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:11:53.187736 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.178379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:37401 (local address 127.5.209.66:35905)
0808 02:11:53.179242 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.179300 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.181328 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.182000 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.182009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.182096 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:53.182605 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.182619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.183591 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.183594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.184497 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.184508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.185034 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.185038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.185216 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.185980 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:53.185982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:53.186041 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:53.186110 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:53.186419 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:53.186432 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:53.186439 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.186576 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.186892 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":183,"threads_started":1}
I20260808 02:11:53.188949 32690 heartbeater.cc:347] Connected to a master server at 127.5.209.126:37401
I20260808 02:11:53.189249 32690 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:53.190059 32690 heartbeater.cc:511] Master 127.5.209.126:37401 requested a full tablet report, sending...
I20260808 02:11:53.191334 32288 ts_manager.cc:194] Registered new tserver with Master: e32d5bb786714e4d998a6efb9ff6b075 (127.5.209.66:44663)
I20260808 02:11:53.191913 32288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.66:35905
W20260808 02:11:53.302299 32695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:53.302567 32695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:53.302606 32695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:53.306725 32695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:53.306770 32695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:53.306854 32695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.209.67
I20260808 02:11:53.311538 32695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.209.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.209.67
--webserver_port=0
--tserver_master_addrs=127.5.209.126:37401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.209.67
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:53.312660 32695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:53.314074 32695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:53.324008 32701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:53.324041 32702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:53.324231 32695 server_base.cc:1034] running on GCE node
W20260808 02:11:53.324222 32706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:53.324765 32695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:53.325434 32695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:53.326620 32695 hybrid_clock.cc:648] HybridClock initialized: now 1786155113326602 us; error 34 us; skew 500 ppm
I20260808 02:11:53.328545 32695 webserver.cc:460] Webserver started at http://127.5.209.67:45325/ using document root <none> and password file <none>
I20260808 02:11:53.329185 32695 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:53.329279 32695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:53.329540 32695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:53.331588 32695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data/instance:
uuid: "48f73d9c46394359821146ca889e96a5"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "28cfd8bed5fb83544b3dc8e4d20cbc28"
server_key_iv: "ae690d84eef2ba7d5860416868f3a608"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.332175 32695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance:
uuid: "48f73d9c46394359821146ca889e96a5"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "28cfd8bed5fb83544b3dc8e4d20cbc28"
server_key_iv: "ae690d84eef2ba7d5860416868f3a608"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.335942 32695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 02:11:53.338784 32712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.339970 32695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:11:53.340090 32695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal
uuid: "48f73d9c46394359821146ca889e96a5"
format_stamp: "Formatted at 2026-08-08 02:11:53 on dist-test-slave-bw88"
server_key: "28cfd8bed5fb83544b3dc8e4d20cbc28"
server_key_iv: "ae690d84eef2ba7d5860416868f3a608"
server_key_version: "encryptionkey@0"
I20260808 02:11:53.340176 32695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:53.361519 32695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:53.362178 32695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:53.362334 32695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:53.362874 32695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:11:53.363806 32695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:11:53.363854 32695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.363894 32695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:11:53.363914 32695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:53.372887 32695 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.67:36057
I20260808 02:11:53.372943 381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.67:36057 every 8 connection(s)
I20260808 02:11:53.373839 32695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/data/info.pb
I20260808 02:11:53.377673 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 32695
I20260808 02:11:53.377815 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal/instance
I20260808 02:11:53.378202 5957 external_mini_cluster.cc:1428] Setting key 02e5f294ffd1a97e6117e2cef8269602
I20260808 02:11:53.384074 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.375727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.67:54865 (local address 127.5.209.126:37401)
0808 02:11:53.375881 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.375885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.376934 (+ 1049us) server_negotiation.cc:407] Connection header received
0808 02:11:53.378811 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.378813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.378863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.378933 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:53.380197 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.380835 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.382100 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.382319 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.382403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.383506 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:53.383509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:53.383511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:53.383522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:53.383538 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:53.383563 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:53.383618 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.383803 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.383877 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:11:53.385125 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.376128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:37401 (local address 127.5.209.67:54865)
0808 02:11:53.376784 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.376825 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.378561 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.379307 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.379317 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.379429 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:11:53.380045 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.380059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.380958 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.380961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.381970 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.381980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.382455 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.382458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.382612 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.383271 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:11:53.383273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:11:53.383312 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:11:53.383377 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:11:53.383637 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:11:53.383644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:11:53.383649 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.383793 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.384136 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260808 02:11:53.386292 382 heartbeater.cc:347] Connected to a master server at 127.5.209.126:37401
I20260808 02:11:53.386564 382 heartbeater.cc:464] Registering TS with master...
I20260808 02:11:53.387137 382 heartbeater.cc:511] Master 127.5.209.126:37401 requested a full tablet report, sending...
I20260808 02:11:53.388260 32288 ts_manager.cc:194] Registered new tserver with Master: 48f73d9c46394359821146ca889e96a5 (127.5.209.67:36057)
I20260808 02:11:53.388841 32288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.209.67:54865
I20260808 02:11:53.392136 5957 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:11:53.398164 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.394205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.5.209.126:37401)
0808 02:11:53.394367 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.394372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.394575 (+ 203us) server_negotiation.cc:407] Connection header received
0808 02:11:53.394730 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.394733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.394784 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.394872 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:53.395158 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.395659 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.396525 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.396735 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.396841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.397691 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:53.397695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:53.397697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:53.397716 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:53.397735 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:53.397743 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:53.397821 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.397945 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.398003 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:11:53.400930 32288 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:11:53.404049 32288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:11:53.418766 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.413179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.5.209.66:44663)
0808 02:11:53.414178 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.414191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.414214 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:11:53.414291 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.414297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.414483 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.414634 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:53.414890 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.416150 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.417549 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418072 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418203 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.418407 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.418512 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.418613 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":155,"threads_started":1}
I20260808 02:11:53.418768 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.412452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:36057 (local address 127.0.0.1:55396)
0808 02:11:53.412983 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.413031 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.413177 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.414126 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.414132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.414173 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:53.414309 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.414316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.415766 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.415770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.417115 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.417125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418244 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.418339 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.418362 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.418435 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.418584 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":78,"spinlock_wait_cycles":5120,"thread_start_us":96,"threads_started":1}
I20260808 02:11:53.418759 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.412627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.66:44663 (local address 127.0.0.1:34222)
0808 02:11:53.413779 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.413794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.413880 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.414619 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.414623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.414642 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:53.414742 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.414752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.416281 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.416284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.417445 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.417456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418203 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.418313 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.418351 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.418462 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.418591 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":35,"thread_start_us":110,"threads_started":1}
I20260808 02:11:53.419392 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.412629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.5.209.67:36057)
0808 02:11:53.413579 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.413592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.413619 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:11:53.413702 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.413710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.413936 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.414116 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:53.414432 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.415642 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.417242 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418092 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418835 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.419062 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.419117 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.419234 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":147,"threads_started":1}
I20260808 02:11:53.419719 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.413188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:34283 (local address 127.0.0.1:39110)
0808 02:11:53.414057 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.414072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.414175 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.414798 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.414803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.414823 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:53.414909 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.414914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.415957 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.415961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.417305 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.417316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418558 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.418645 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.418669 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.418740 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.418810 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":374,"spinlock_wait_cycles":11776,"thread_start_us":101,"threads_started":1}
I20260808 02:11:53.420222 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.413449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.5.209.65:34283)
0808 02:11:53.413887 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.413898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.414272 (+ 374us) server_negotiation.cc:407] Connection header received
0808 02:11:53.414353 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.414357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.414576 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.414727 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:53.415019 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.415834 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.417446 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.418203 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.418298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.418484 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.419429 (+ 945us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.420004 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
I20260808 02:11:53.422082 32611 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 (DEFAULT_TABLE table=test-table [id=b7327ad9f6824831850c367e2cbe4595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:53.422603 303 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 (DEFAULT_TABLE table=test-table [id=b7327ad9f6824831850c367e2cbe4595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:53.423121 32611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2e1400e509429d92bb770d9b9118d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:53.423235 32449 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 (DEFAULT_TABLE table=test-table [id=b7327ad9f6824831850c367e2cbe4595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:11:53.423696 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2e1400e509429d92bb770d9b9118d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:53.424255 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2e1400e509429d92bb770d9b9118d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:53.430089 404 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Bootstrap starting.
I20260808 02:11:53.431598 407 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Bootstrap starting.
I20260808 02:11:53.432452 404 tablet_bootstrap.cc:654] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:53.433219 404 log.cc:824] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:53.433449 407 tablet_bootstrap.cc:654] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:53.434329 407 log.cc:824] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:53.435020 405 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Bootstrap starting.
I20260808 02:11:53.435637 404 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: No bootstrap required, opened a new log
I20260808 02:11:53.435855 404 ts_tablet_manager.cc:1397] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260808 02:11:53.437211 405 tablet_bootstrap.cc:654] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:53.438066 405 log.cc:824] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:53.438506 407 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: No bootstrap required, opened a new log
I20260808 02:11:53.438710 407 ts_tablet_manager.cc:1397] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260808 02:11:53.440430 405 tablet_bootstrap.cc:492] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: No bootstrap required, opened a new log
I20260808 02:11:53.440388 404 raft_consensus.cc:348] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.440670 404 raft_consensus.cc:374] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:53.440726 404 raft_consensus.cc:729] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32d5bb786714e4d998a6efb9ff6b075, State: Initialized, Role: FOLLOWER
I20260808 02:11:53.440836 405 ts_tablet_manager.cc:1397] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260808 02:11:53.441258 404 consensus_queue.cc:260] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.442198 32690 heartbeater.cc:503] Master 127.5.209.126:37401 was elected leader, sending a full tablet report...
I20260808 02:11:53.442375 404 ts_tablet_manager.cc:1428] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260808 02:11:53.443796 407 raft_consensus.cc:348] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.444120 407 raft_consensus.cc:374] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:53.444243 407 raft_consensus.cc:729] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fe3aa92016405cb694510c0aa10c31, State: Initialized, Role: FOLLOWER
I20260808 02:11:53.444708 407 consensus_queue.cc:260] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.445396 32534 heartbeater.cc:503] Master 127.5.209.126:37401 was elected leader, sending a full tablet report...
I20260808 02:11:53.445786 407 ts_tablet_manager.cc:1428] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260808 02:11:53.446569 405 raft_consensus.cc:348] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.446792 405 raft_consensus.cc:374] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:53.446862 405 raft_consensus.cc:729] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f73d9c46394359821146ca889e96a5, State: Initialized, Role: FOLLOWER
I20260808 02:11:53.447335 405 consensus_queue.cc:260] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.448033 382 heartbeater.cc:503] Master 127.5.209.126:37401 was elected leader, sending a full tablet report...
I20260808 02:11:53.448642 405 ts_tablet_manager.cc:1428] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
W20260808 02:11:53.472421 32535 tablet.cc:2369] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:11:53.625092 384 tablet.cc:2369] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:53.654304 412 raft_consensus.cc:482] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:11:53.654531 412 raft_consensus.cc:504] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.655797 412 leader_election.cc:288] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48f73d9c46394359821146ca889e96a5 (127.5.209.67:36057), 08fe3aa92016405cb694510c0aa10c31 (127.5.209.65:34283)
I20260808 02:11:53.661012 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.656019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.67:36057 (local address 127.5.209.66:45075)
0808 02:11:53.656166 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.656184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.656289 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.656652 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.656656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.656684 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:53.656872 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.656882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.658081 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.658084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.659508 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.659520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.660436 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.660440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.660594 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.660636 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.660721 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.660839 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":114}
I20260808 02:11:53.661018 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.656183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:45075 (local address 127.5.209.67:36057)
0808 02:11:53.656377 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.656383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.656398 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:11:53.656458 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.656461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.656520 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.656605 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:53.657016 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.657964 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.659683 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.660310 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.660436 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.660489 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.660791 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.660878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 02:11:53.661018 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.656371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.209.66:33543 (local address 127.5.209.65:34283)
0808 02:11:53.656561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.656565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.656606 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:11:53.656669 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.656672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.656724 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.656829 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:11:53.657185 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.658037 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.659683 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.660296 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.660436 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.660489 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.660791 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.660878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 02:11:53.661021 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.656183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:34283 (local address 127.5.209.66:33543)
0808 02:11:53.656489 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.656501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 02:11:53.656592 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:11:53.656870 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:11:53.656875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:11:53.656898 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:11:53.657045 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.657052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.658147 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.658149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.659508 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:11:53.659520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.660436 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.660440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:11:53.660578 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.660650 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 02:11:53.660721 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:11:53.660839 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":143,"thread_start_us":109,"threads_started":1}
I20260808 02:11:53.661845 32472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2e1400e509429d92bb770d9b9118d1" candidate_uuid: "e32d5bb786714e4d998a6efb9ff6b075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fe3aa92016405cb694510c0aa10c31" is_pre_election: true
I20260808 02:11:53.661854 326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2e1400e509429d92bb770d9b9118d1" candidate_uuid: "e32d5bb786714e4d998a6efb9ff6b075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f73d9c46394359821146ca889e96a5" is_pre_election: true
I20260808 02:11:53.662212 326 raft_consensus.cc:2454] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32d5bb786714e4d998a6efb9ff6b075 in term 0.
I20260808 02:11:53.662212 32472 raft_consensus.cc:2454] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e32d5bb786714e4d998a6efb9ff6b075 in term 0.
I20260808 02:11:53.662778 32556 leader_election.cc:302] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08fe3aa92016405cb694510c0aa10c31, 48f73d9c46394359821146ca889e96a5, e32d5bb786714e4d998a6efb9ff6b075; no voters:
I20260808 02:11:53.663090 412 raft_consensus.cc:2781] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:11:53.663170 412 raft_consensus.cc:482] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:11:53.663201 412 raft_consensus.cc:3037] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:53.664281 412 raft_consensus.cc:504] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.664683 412 leader_election.cc:288] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [CANDIDATE]: Term 1 election: Requested vote from peers 48f73d9c46394359821146ca889e96a5 (127.5.209.67:36057), 08fe3aa92016405cb694510c0aa10c31 (127.5.209.65:34283)
I20260808 02:11:53.665010 326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2e1400e509429d92bb770d9b9118d1" candidate_uuid: "e32d5bb786714e4d998a6efb9ff6b075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f73d9c46394359821146ca889e96a5"
I20260808 02:11:53.665132 32472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2e1400e509429d92bb770d9b9118d1" candidate_uuid: "e32d5bb786714e4d998a6efb9ff6b075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fe3aa92016405cb694510c0aa10c31"
I20260808 02:11:53.665136 326 raft_consensus.cc:3037] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:53.665278 32472 raft_consensus.cc:3037] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:53.666296 326 raft_consensus.cc:2454] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32d5bb786714e4d998a6efb9ff6b075 in term 1.
I20260808 02:11:53.666499 32472 raft_consensus.cc:2454] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32d5bb786714e4d998a6efb9ff6b075 in term 1.
I20260808 02:11:53.666682 32554 leader_election.cc:302] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f73d9c46394359821146ca889e96a5, e32d5bb786714e4d998a6efb9ff6b075; no voters:
I20260808 02:11:53.666867 412 raft_consensus.cc:2781] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:53.667086 412 raft_consensus.cc:686] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 1 LEADER]: Becoming Leader. State: Replica: e32d5bb786714e4d998a6efb9ff6b075, State: Running, Role: LEADER
I20260808 02:11:53.667425 412 consensus_queue.cc:237] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } }
I20260808 02:11:53.670928 32288 catalog_manager.cc:5495] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 reported cstate change: term changed from 0 to 1, leader changed from <none> to e32d5bb786714e4d998a6efb9ff6b075 (127.5.209.66). New cstate: current_term: 1 leader_uuid: "e32d5bb786714e4d998a6efb9ff6b075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32d5bb786714e4d998a6efb9ff6b075" member_type: VOTER last_known_addr { host: "127.5.209.66" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 } health_report { overall_health: UNKNOWN } } }
W20260808 02:11:53.676831 32692 tablet.cc:2369] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:11:53.712985 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.708240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.5.209.66:44663)
0808 02:11:53.708429 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:53.708436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:53.708489 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:11:53.708598 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:53.708602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:53.708685 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:53.708804 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:11:53.709106 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.710051 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.710998 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:53.711410 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:53.711540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:53.711887 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:11:53.712477 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:11:53.712621 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:53.712710 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:11:53.712777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:11:53.719962 326 raft_consensus.cc:1261] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 1 FOLLOWER]: Refusing update from remote peer e32d5bb786714e4d998a6efb9ff6b075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:53.720685 32472 raft_consensus.cc:1261] T 1b2e1400e509429d92bb770d9b9118d1 P 08fe3aa92016405cb694510c0aa10c31 [term 1 FOLLOWER]: Refusing update from remote peer e32d5bb786714e4d998a6efb9ff6b075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:11:53.720844 412 consensus_queue.cc:1035] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f73d9c46394359821146ca889e96a5" member_type: VOTER last_known_addr { host: "127.5.209.67" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:53.721431 417 consensus_queue.cc:1035] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08fe3aa92016405cb694510c0aa10c31" member_type: VOTER last_known_addr { host: "127.5.209.65" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:11:53.732033 421 mvcc.cc:205] Tried to move back new op lower bound from 7316091345788452864 to 7316091345584492544. Current Snapshot: MvccSnapshot[applied={T|T < 7316091345788452864}]
I20260808 02:11:53.732813 422 mvcc.cc:205] Tried to move back new op lower bound from 7316091345788452864 to 7316091345584492544. Current Snapshot: MvccSnapshot[applied={T|T < 7316091345788452864}]
I20260808 02:11:53.739183 32288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:53.739344 32288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:11:53.742149 32288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be87051ae95148139d015214d7e473e1: Sending DeleteTablet for 3 replicas of tablet 1b2e1400e509429d92bb770d9b9118d1
I20260808 02:11:53.743098 303 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:53 UTC) from {username='slave'} at 127.0.0.1:55396
I20260808 02:11:53.743110 32449 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:53 UTC) from {username='slave'} at 127.0.0.1:39110
I20260808 02:11:53.743150 32611 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2e1400e509429d92bb770d9b9118d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:11:53 UTC) from {username='slave'} at 127.0.0.1:34222
I20260808 02:11:53.743765 434 tablet_replica.cc:330] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: stopping tablet replica
I20260808 02:11:53.743788 433 tablet_replica.cc:330] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: stopping tablet replica
I20260808 02:11:53.744128 433 raft_consensus.cc:2229] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:11:53.744129 434 raft_consensus.cc:2229] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:11:53.744462 434 raft_consensus.cc:2258] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:53.744513 433 raft_consensus.cc:2258] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:11:53.744796 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 32365
I20260808 02:11:53.746455 433 ts_tablet_manager.cc:1905] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:53.748283 434 ts_tablet_manager.cc:1905] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:11:53.749439 433 ts_tablet_manager.cc:1918] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:53.749533 433 log.cc:1196] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-1/wal/wals/1b2e1400e509429d92bb770d9b9118d1
I20260808 02:11:53.749863 433 ts_tablet_manager.cc:1939] T 1b2e1400e509429d92bb770d9b9118d1 P e32d5bb786714e4d998a6efb9ff6b075: Deleting consensus metadata
I20260808 02:11:53.751149 434 ts_tablet_manager.cc:1918] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:11:53.751245 434 log.cc:1196] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Deleting WAL directory at /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786154850122309-5957-0/minicluster-data/ts-2/wal/wals/1b2e1400e509429d92bb770d9b9118d1
I20260808 02:11:53.751226 32270 catalog_manager.cc:4841] TS e32d5bb786714e4d998a6efb9ff6b075 (127.5.209.66:44663): tablet 1b2e1400e509429d92bb770d9b9118d1 (table test-table [id=b7327ad9f6824831850c367e2cbe4595]) successfully deleted
I20260808 02:11:53.751583 434 ts_tablet_manager.cc:1939] T 1b2e1400e509429d92bb770d9b9118d1 P 48f73d9c46394359821146ca889e96a5: Deleting consensus metadata
I20260808 02:11:53.752662 32268 catalog_manager.cc:4841] TS 48f73d9c46394359821146ca889e96a5 (127.5.209.67:36057): tablet 1b2e1400e509429d92bb770d9b9118d1 (table test-table [id=b7327ad9f6824831850c367e2cbe4595]) successfully deleted
W20260808 02:11:53.762383 32270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.209.65:34283 (error 108)
I20260808 02:11:53.762452 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 32537
I20260808 02:11:53.765511 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:53.765075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.65:34283 (local address 127.0.0.1:39112)
0808 02:11:53.765301 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 02:11:53.765395 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.65:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260808 02:11:53.765704 32270 catalog_manager.cc:4584] TS 08fe3aa92016405cb694510c0aa10c31 (127.5.209.65:34283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b2e1400e509429d92bb770d9b9118d1: Network error: Client connection negotiation failed: client connection to 127.5.209.65:34283: connect: Connection refused (error 111)
I20260808 02:11:53.776046 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 32695
I20260808 02:11:53.789388 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 32251
2026-08-08T02:11:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1302 ms)
[----------] 6 tests from EncryptionPolicyTest (11299 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:53 dist-test-slave-bw88 krb5kdc[441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:53 dist-test-slave-bw88 krb5kdc[441](info): set up 2 sockets
Aug 08 02:11:53 dist-test-slave-bw88 krb5kdc[441](info): commencing operation
krb5kdc: starting...
W20260808 02:11:55.007247 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:11:55.148541 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:55.266040 457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:55.266345 457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:55.266425 457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:55.270462 457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:55.270550 457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:55.270589 457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:55.270627 457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:55.270660 457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:55.276067 457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:55.277231 457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:55.278460 457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:55.287125 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:55.287317 457 server_base.cc:1034] running on GCE node
W20260808 02:11:55.287135 462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:55.287125 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:55.288017 457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:55.289110 457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:55.290290 457 hybrid_clock.cc:648] HybridClock initialized: now 1786155115290246 us; error 55 us; skew 500 ppm
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:55.293172 457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:11:55.294162 457 webserver.cc:460] Webserver started at http://127.0.0.1:43847/ using document root <none> and password file <none>
I20260808 02:11:55.294720 457 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:55.294780 457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:55.294947 457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:55.297060 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "b77c834ad47d4387b6580dd023501339"
format_stamp: "Formatted at 2026-08-08 02:11:55 on dist-test-slave-bw88"
server_key: "a00a08117b1d5bd67e39628386ef552e"
server_key_iv: "b1d5fb708437151e3ae851f4ad88e0b4"
server_key_version: "encryptionkey@0"
I20260808 02:11:55.297680 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "b77c834ad47d4387b6580dd023501339"
format_stamp: "Formatted at 2026-08-08 02:11:55 on dist-test-slave-bw88"
server_key: "a00a08117b1d5bd67e39628386ef552e"
server_key_iv: "b1d5fb708437151e3ae851f4ad88e0b4"
server_key_version: "encryptionkey@0"
I20260808 02:11:55.301560 457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:11:55.304046 474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:55.305409 457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:11:55.305606 457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "b77c834ad47d4387b6580dd023501339"
format_stamp: "Formatted at 2026-08-08 02:11:55 on dist-test-slave-bw88"
server_key: "a00a08117b1d5bd67e39628386ef552e"
server_key_iv: "b1d5fb708437151e3ae851f4ad88e0b4"
server_key_version: "encryptionkey@0"
I20260808 02:11:55.305724 457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:55.331250 457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:55.334774 457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:55.335009 457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:55.343489 457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41437
I20260808 02:11:55.343518 536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41437 every 8 connection(s)
I20260808 02:11:55.344595 457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:55.346808 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 457
I20260808 02:11:55.346982 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:55.347350 5957 external_mini_cluster.cc:1428] Setting key 8a20223b513771fc541348a9acc57f04
I20260808 02:11:55.347743 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:55.354312 537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339: Bootstrap starting.
I20260808 02:11:55.356922 537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339: Neither blocks nor log segments found. Creating new log.
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:11:55.357920 537 log.cc:824] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:55.360713 537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339: No bootstrap required, opened a new log
I20260808 02:11:55.363183 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:55.348930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39817 (local address 127.0.0.1:41437)
0808 02:11:55.349513 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:55.349532 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:55.349575 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:11:55.350624 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:55.350644 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:55.351016 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:55.351391 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:55.352256 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:55.353250 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:55.354160 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:55.354698 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:55.354870 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:55.357877 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:55.357882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:55.357897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:55.357924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:55.359877 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:55.360538 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:55.360544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:55.360548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:55.360665 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:55.361062 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:55.361066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:55.361068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:55.361610 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:55.361782 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:55.362135 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:11:55.362392 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260808 02:11:55.365049 537 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260808 02:11:55.365312 537 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:55.365396 537 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77c834ad47d4387b6580dd023501339, State: Initialized, Role: FOLLOWER
I20260808 02:11:55.365897 537 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260808 02:11:55.366067 537 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:55.366142 537 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:55.366253 537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:55.366423 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:11:55.367489 537 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260808 02:11:55.367894 537 leader_election.cc:302] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77c834ad47d4387b6580dd023501339; no voters:
I20260808 02:11:55.368268 537 leader_election.cc:288] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:55.368373 551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:55.368696 551 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [term 1 LEADER]: Becoming Leader. State: Replica: b77c834ad47d4387b6580dd023501339, State: Running, Role: LEADER
I20260808 02:11:55.369055 551 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260808 02:11:55.369648 537 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:55.370893 553 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77c834ad47d4387b6580dd023501339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260808 02:11:55.371027 553 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:55.371335 552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77c834ad47d4387b6580dd023501339. Latest consensus state: current_term: 1 leader_uuid: "b77c834ad47d4387b6580dd023501339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77c834ad47d4387b6580dd023501339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260808 02:11:55.371452 552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:55.371415 561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:55.375151 561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:55.380388 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:55.368942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57725 (local address 127.0.0.1:41437)
0808 02:11:55.369044 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:55.369048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:55.371993 (+ 2945us) server_negotiation.cc:407] Connection header received
0808 02:11:55.372160 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:55.372163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:55.372220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:55.372300 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:55.373137 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:55.373746 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:55.374582 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:55.374820 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:55.374935 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:55.376363 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:55.376367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:55.376370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:11:55.376405 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:55.378146 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:55.378742 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:55.378748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:55.378751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:55.378864 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:11:55.379278 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:11:55.379282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:11:55.379284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:11:55.379522 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:55.379670 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:55.380119 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 02:11:55.380227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:11:55.381369 561 catalog_manager.cc:1347] Generated new cluster ID: 09c45843af43494d9ec03474595bc9d6
I20260808 02:11:55.381433 561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:55.397065 561 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:55.398147 561 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:55.413694 561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77c834ad47d4387b6580dd023501339: Generated new TSK 0
I20260808 02:11:55.414479 561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:55.425272 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 457
W20260808 02:11:55.439337 438 connection.cc:535] client connection to 127.0.0.1:41437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41437): Connection reset by peer (error 104)
2026-08-08T02:11:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[575](info): set up 2 sockets
Aug 08 02:11:55 dist-test-slave-bw88 krb5kdc[575](info): commencing operation
krb5kdc: starting...
W20260808 02:11:56.626197 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:56 dist-test-slave-bw88 krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:11:56.760439 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:56.882532 593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:56.882864 593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:56.882951 593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:56.887151 593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:56.887254 593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:56.887295 593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:56.887354 593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:56.887396 593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:56.892910 593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:56.894272 593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:56.895412 593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:56.904649 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:56.904632 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:56.904840 593 server_base.cc:1034] running on GCE node
W20260808 02:11:56.904632 598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:56.905678 593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:56.906836 593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:56.908005 593 hybrid_clock.cc:648] HybridClock initialized: now 1786155116907967 us; error 49 us; skew 500 ppm
Aug 08 02:11:56 dist-test-slave-bw88 krb5kdc[575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:56.911095 593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:11:56.912103 593 webserver.cc:460] Webserver started at http://127.0.0.1:43011/ using document root <none> and password file <none>
I20260808 02:11:56.912729 593 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:56.912796 593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:56.913056 593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:56.915298 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "7b30e61d75194ab69cb60243b17a5ab4"
format_stamp: "Formatted at 2026-08-08 02:11:56 on dist-test-slave-bw88"
server_key: "4d1008cc0e8f2d2632ce21b691e56497"
server_key_iv: "69e650abeb20e0de6ce94c4aafdc0236"
server_key_version: "encryptionkey@0"
I20260808 02:11:56.915923 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "7b30e61d75194ab69cb60243b17a5ab4"
format_stamp: "Formatted at 2026-08-08 02:11:56 on dist-test-slave-bw88"
server_key: "4d1008cc0e8f2d2632ce21b691e56497"
server_key_iv: "69e650abeb20e0de6ce94c4aafdc0236"
server_key_version: "encryptionkey@0"
I20260808 02:11:56.919942 593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:11:56.922561 610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:56.923825 593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 02:11:56.923986 593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "7b30e61d75194ab69cb60243b17a5ab4"
format_stamp: "Formatted at 2026-08-08 02:11:56 on dist-test-slave-bw88"
server_key: "4d1008cc0e8f2d2632ce21b691e56497"
server_key_iv: "69e650abeb20e0de6ce94c4aafdc0236"
server_key_version: "encryptionkey@0"
I20260808 02:11:56.924108 593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:56.943243 593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:56.946328 593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:56.946578 593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:56.954270 593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45155
I20260808 02:11:56.954298 677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45155 every 8 connection(s)
I20260808 02:11:56.955276 593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:56.958257 678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:56.958567 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 593
I20260808 02:11:56.958745 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:56.959067 5957 external_mini_cluster.cc:1428] Setting key 673a22e624a5070c18e40b9cbbcf4ebd
I20260808 02:11:56.965922 678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4: Bootstrap starting.
I20260808 02:11:56.968753 678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:56.969717 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:56.960563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59047 (local address 127.0.0.1:45155)
0808 02:11:56.961055 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:56.961076 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:56.961125 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:11:56.961954 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:56.961981 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:56.962432 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:56.962841 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:56.963288 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:56.964879 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:56.965901 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:56.966585 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:56.966743 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:56.967632 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:56.967639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:56.967654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:56.968149 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:56.968218 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:56.968248 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:56.968385 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:56.968572 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 02:11:56.968717 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260808 02:11:56.969682 678 log.cc:824] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:56.972047 678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4: No bootstrap required, opened a new log
I20260808 02:11:56.972682 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:11:56.976183 678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260808 02:11:56.976442 678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:56.976526 678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b30e61d75194ab69cb60243b17a5ab4, State: Initialized, Role: FOLLOWER
I20260808 02:11:56.977020 678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260808 02:11:56.977183 678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:56.977277 678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:56.977401 678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:56.978359 678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260808 02:11:56.978747 678 leader_election.cc:302] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b30e61d75194ab69cb60243b17a5ab4; no voters:
I20260808 02:11:56.979125 678 leader_election.cc:288] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:56.979249 692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:56.979408 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:56.975106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47903 (local address 127.0.0.1:45155)
0808 02:11:56.975243 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:56.975249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:56.975430 (+ 181us) server_negotiation.cc:407] Connection header received
0808 02:11:56.975554 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:56.975556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:56.975594 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:56.975661 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:56.975991 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:56.976552 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:56.977583 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:11:56.977882 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:11:56.977992 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:11:56.978868 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:56.978873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:56.978878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:56.978893 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:56.978921 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:56.978933 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:56.979042 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:56.979169 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:11:56.979202 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:11:56.979628 692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b30e61d75194ab69cb60243b17a5ab4, State: Running, Role: LEADER
I20260808 02:11:56.979935 692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260808 02:11:56.980530 678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:56.981197 692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b30e61d75194ab69cb60243b17a5ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } } }
I20260808 02:11:56.981365 692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:56.981607 692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b30e61d75194ab69cb60243b17a5ab4. Latest consensus state: current_term: 1 leader_uuid: "7b30e61d75194ab69cb60243b17a5ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b30e61d75194ab69cb60243b17a5ab4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } } }
I20260808 02:11:56.981714 692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:56.981880 702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:56.985252 702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:56.991875 702 catalog_manager.cc:1347] Generated new cluster ID: 62759ae84b8c4a448920e9558bf56d4d
I20260808 02:11:56.991969 702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:57.014506 702 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:57.015614 702 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:57.023392 702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b30e61d75194ab69cb60243b17a5ab4: Generated new TSK 0
I20260808 02:11:57.024142 702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:57.032409 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 593
W20260808 02:11:57.048200 573 connection.cc:535] client connection to 127.0.0.1:45155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45155): Connection reset by peer (error 104)
2026-08-08T02:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:57 dist-test-slave-bw88 krb5kdc[720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:57 dist-test-slave-bw88 krb5kdc[720](info): set up 2 sockets
Aug 08 02:11:57 dist-test-slave-bw88 krb5kdc[720](info): commencing operation
krb5kdc: starting...
W20260808 02:11:58.265210 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:58 dist-test-slave-bw88 krb5kdc[720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:11:58.400024 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:11:58.515547 739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:11:58.515810 739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:11:58.515852 739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:11:58.520809 739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:11:58.520876 739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:11:58.520896 739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:11:58.520912 739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:11:58.520927 739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:11:58.526119 739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:11:58.527209 739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:11:58.528258 739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:11:58.537143 746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:58.537241 739 server_base.cc:1034] running on GCE node
W20260808 02:11:58.537137 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:11:58.537137 751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:11:58.538033 739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:11:58.539183 739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:11:58.540412 739 hybrid_clock.cc:648] HybridClock initialized: now 1786155118540381 us; error 40 us; skew 500 ppm
Aug 08 02:11:58 dist-test-slave-bw88 krb5kdc[720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:11:58.547070 739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:11:58.548036 739 webserver.cc:460] Webserver started at http://127.0.0.1:44039/ using document root <none> and password file <none>
I20260808 02:11:58.548596 739 fs_manager.cc:359] Metadata directory not provided
I20260808 02:11:58.548658 739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:11:58.548832 739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:11:58.550956 739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "77fe1ea56946489f996c69485f7efb43"
format_stamp: "Formatted at 2026-08-08 02:11:58 on dist-test-slave-bw88"
server_key: "ab0aef47ff0074fc93d816f558196fcd"
server_key_iv: "877707ca8e22e9dd48da97bf78dbe936"
server_key_version: "encryptionkey@0"
I20260808 02:11:58.551481 739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "77fe1ea56946489f996c69485f7efb43"
format_stamp: "Formatted at 2026-08-08 02:11:58 on dist-test-slave-bw88"
server_key: "ab0aef47ff0074fc93d816f558196fcd"
server_key_iv: "877707ca8e22e9dd48da97bf78dbe936"
server_key_version: "encryptionkey@0"
I20260808 02:11:58.555256 739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 02:11:58.557680 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:11:58.558898 739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:11:58.559049 739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "77fe1ea56946489f996c69485f7efb43"
format_stamp: "Formatted at 2026-08-08 02:11:58 on dist-test-slave-bw88"
server_key: "ab0aef47ff0074fc93d816f558196fcd"
server_key_iv: "877707ca8e22e9dd48da97bf78dbe936"
server_key_version: "encryptionkey@0"
I20260808 02:11:58.559150 739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:11:58.563990 739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:11:58.566915 739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:11:58.567116 739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:11:58.574450 739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35897
I20260808 02:11:58.574460 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35897 every 8 connection(s)
I20260808 02:11:58.575400 739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:11:58.577136 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 739
I20260808 02:11:58.577301 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:11:58.577683 5957 external_mini_cluster.cc:1428] Setting key 8120c56dd52a5ed6b9f23cdf723345e7
I20260808 02:11:58.578800 828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:11:58.583019 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:58.579089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39899 (local address 127.0.0.1:35897)
0808 02:11:58.579547 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:58.579563 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:58.579601 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:11:58.580321 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:58.580340 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:58.580629 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:58.580967 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:58.581833 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:58.581841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:58.581860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:58.581871 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:58.581926 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:58.581945 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:58.582109 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:58.582183 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 02:11:58.582341 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":144,"threads_started":1}
I20260808 02:11:58.585202 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43: Bootstrap starting.
I20260808 02:11:58.586115 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:11:58.587762 828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43: Neither blocks nor log segments found. Creating new log.
I20260808 02:11:58.588686 828 log.cc:824] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43: Log is configured to *not* fsync() on all Append() calls
I20260808 02:11:58.589947 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:11:58.588292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34663 (local address 127.0.0.1:35897)
0808 02:11:58.588426 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 02:11:58.588430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:11:58.588587 (+ 157us) server_negotiation.cc:407] Connection header received
0808 02:11:58.588686 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:11:58.588689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:11:58.588733 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:11:58.588814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:11:58.589502 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:11:58.589505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:11:58.589507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:11:58.589518 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:11:58.589543 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:11:58.589561 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:11:58.589629 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:11:58.589747 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 02:11:58.589807 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:11:58.591243 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43: No bootstrap required, opened a new log
I20260808 02:11:58.594796 828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } }
I20260808 02:11:58.595023 828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:11:58.595084 828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fe1ea56946489f996c69485f7efb43, State: Initialized, Role: FOLLOWER
I20260808 02:11:58.595526 828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } }
I20260808 02:11:58.595652 828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:11:58.595698 828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:11:58.595769 828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:11:58.596865 828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } }
I20260808 02:11:58.597258 828 leader_election.cc:302] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77fe1ea56946489f996c69485f7efb43; no voters:
I20260808 02:11:58.597633 828 leader_election.cc:288] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:11:58.597767 843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:11:58.598024 843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [term 1 LEADER]: Becoming Leader. State: Replica: 77fe1ea56946489f996c69485f7efb43, State: Running, Role: LEADER
I20260808 02:11:58.598357 843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } }
I20260808 02:11:58.598938 828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:11:58.600087 845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77fe1ea56946489f996c69485f7efb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } } }
I20260808 02:11:58.600247 845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:58.600540 846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fe1ea56946489f996c69485f7efb43. Latest consensus state: current_term: 1 leader_uuid: "77fe1ea56946489f996c69485f7efb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fe1ea56946489f996c69485f7efb43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35897 } } }
I20260808 02:11:58.600643 855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:11:58.600661 846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43 [sys.catalog]: This master's current role is: LEADER
I20260808 02:11:58.604121 855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:11:58.609635 855 catalog_manager.cc:1347] Generated new cluster ID: 879eaf02e40441498139cd81f413821c
I20260808 02:11:58.609705 855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:11:58.618697 855 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:11:58.619057 855 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:11:58.628964 855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77fe1ea56946489f996c69485f7efb43: Generated new TSK 0
I20260808 02:11:58.629861 855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:11:58.638803 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 739
2026-08-08T02:11:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1601 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:11:58 dist-test-slave-bw88 krb5kdc[867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:11:58 dist-test-slave-bw88 krb5kdc[867](info): set up 2 sockets
Aug 08 02:11:58 dist-test-slave-bw88 krb5kdc[867](info): commencing operation
krb5kdc: starting...
W20260808 02:11:59.835736 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:11:59 dist-test-slave-bw88 krb5kdc[867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:11:59.969039 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:00.087173 884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:00.087517 884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:00.087634 884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:00.091696 884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:00.091801 884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:00.091842 884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:00.091879 884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:00.091912 884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:00.097200 884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:00.098384 884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:00.099496 884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:00.108282 894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:00.108299 892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:00.108299 890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:00.108707 884 server_base.cc:1034] running on GCE node
I20260808 02:12:00.109282 884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:00.110473 884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:00.111681 884 hybrid_clock.cc:648] HybridClock initialized: now 1786155120111647 us; error 46 us; skew 500 ppm
Aug 08 02:12:00 dist-test-slave-bw88 krb5kdc[867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:00.114542 884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:00.115520 884 webserver.cc:460] Webserver started at http://127.0.0.1:35409/ using document root <none> and password file <none>
I20260808 02:12:00.116111 884 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:00.116207 884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:00.116454 884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:00.118615 884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "0581528093284acd8310772b074a602b"
format_stamp: "Formatted at 2026-08-08 02:12:00 on dist-test-slave-bw88"
server_key: "b4db1877536feb80cea3b1170f8065f6"
server_key_iv: "27f7098f58b7881d3b1290bec6da74a4"
server_key_version: "encryptionkey@0"
I20260808 02:12:00.119211 884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "0581528093284acd8310772b074a602b"
format_stamp: "Formatted at 2026-08-08 02:12:00 on dist-test-slave-bw88"
server_key: "b4db1877536feb80cea3b1170f8065f6"
server_key_iv: "27f7098f58b7881d3b1290bec6da74a4"
server_key_version: "encryptionkey@0"
I20260808 02:12:00.123129 884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 02:12:00.125741 904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:00.127086 884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:12:00.127259 884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "0581528093284acd8310772b074a602b"
format_stamp: "Formatted at 2026-08-08 02:12:00 on dist-test-slave-bw88"
server_key: "b4db1877536feb80cea3b1170f8065f6"
server_key_iv: "27f7098f58b7881d3b1290bec6da74a4"
server_key_version: "encryptionkey@0"
I20260808 02:12:00.127388 884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:00.138689 884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:00.141706 884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:00.141934 884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:00.149549 884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39929
I20260808 02:12:00.149576 971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39929 every 8 connection(s)
I20260808 02:12:00.150556 884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:00.153417 973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:00.155786 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 884
I20260808 02:12:00.155966 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:00.156302 5957 external_mini_cluster.cc:1428] Setting key 9ef1325d7945c1aae4899b3d25aa4fdc
I20260808 02:12:00.159768 973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: Bootstrap starting.
I20260808 02:12:00.162101 973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:00.162864 973 log.cc:824] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:00.164912 973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: No bootstrap required, opened a new log
Aug 08 02:12:00 dist-test-slave-bw88 krb5kdc[867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:12:00.168893 973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260808 02:12:00.169119 973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:00.169186 973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0581528093284acd8310772b074a602b, State: Initialized, Role: FOLLOWER
I20260808 02:12:00.169739 973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260808 02:12:00.169902 973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:00.169978 973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:00.170116 973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:00.171048 973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260808 02:12:00.171104 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:00.157822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56091 (local address 127.0.0.1:39929)
0808 02:12:00.158277 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:00.158293 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:00.158328 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:12:00.158998 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:00.159015 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:00.159346 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:00.159696 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:00.160510 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:00.161615 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:00.162550 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:00.163074 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:00.163251 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:00.166130 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:00.166135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:00.166149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:00.166177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:00.167983 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:00.168469 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:00.168482 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:00.168487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:00.168628 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:00.168925 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:00.168928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:00.168930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:00.169496 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:00.169740 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:00.170014 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 02:12:00.170352 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260808 02:12:00.171460 973 leader_election.cc:302] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0581528093284acd8310772b074a602b; no voters:
I20260808 02:12:00.171837 973 leader_election.cc:288] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:00.171952 978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:00.172188 978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [term 1 LEADER]: Becoming Leader. State: Replica: 0581528093284acd8310772b074a602b, State: Running, Role: LEADER
I20260808 02:12:00.172508 978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20260808 02:12:00.173125 973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:00.173791 979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0581528093284acd8310772b074a602b. Latest consensus state: current_term: 1 leader_uuid: "0581528093284acd8310772b074a602b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20260808 02:12:00.173990 979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:00.174296 980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0581528093284acd8310772b074a602b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0581528093284acd8310772b074a602b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20260808 02:12:00.174394 980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:00.174553 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 02:12:00.178442 995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:00.178529 995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:00.178591 987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:00.179471 987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:00.185170 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:00.176836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36009 (local address 127.0.0.1:39929)
0808 02:12:00.176936 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:00.176940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:00.177176 (+ 236us) server_negotiation.cc:407] Connection header received
0808 02:12:00.177332 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:00.177377 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:00.177427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:00.177510 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:00.178174 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:00.178789 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:00.179700 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:00.179959 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:00.180083 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:00.181498 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:00.181503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:00.181506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:00.181531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:00.183433 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:00.183911 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:00.183916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:00.183918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:00.184006 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:00.184290 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:00.184294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:00.184296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:00.184479 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:00.184635 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:00.184873 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:12:00.184991 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 02:12:00.185719 987 catalog_manager.cc:1347] Generated new cluster ID: bdde61a497704ca7ac8da752eeff5350
I20260808 02:12:00.185792 987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:00.200034 987 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:00.201529 987 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:00.208840 987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0581528093284acd8310772b074a602b: Generated new TSK 0
I20260808 02:12:00.209654 987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:00.215576 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 884
2026-08-08T02:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:00 dist-test-slave-bw88 krb5kdc[1007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:00 dist-test-slave-bw88 krb5kdc[1007](info): set up 2 sockets
Aug 08 02:12:00 dist-test-slave-bw88 krb5kdc[1007](info): commencing operation
krb5kdc: starting...
W20260808 02:12:01.419133 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:01 dist-test-slave-bw88 krb5kdc[1007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:01.552619 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:01.670683 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:01.670995 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:01.671080 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:01.675105 1024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:01.675195 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:01.675235 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:01.675293 1024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:01.675334 1024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:01.680599 1024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:01.681833 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:01.682904 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:01.691783 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:01.691995 1024 server_base.cc:1034] running on GCE node
W20260808 02:12:01.691787 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:01.691787 1032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:01.692771 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:01.694003 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:01.695206 1024 hybrid_clock.cc:648] HybridClock initialized: now 1786155121695177 us; error 42 us; skew 500 ppm
Aug 08 02:12:01 dist-test-slave-bw88 krb5kdc[1007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:01.698112 1024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:01.699167 1024 webserver.cc:460] Webserver started at http://127.0.0.1:41251/ using document root <none> and password file <none>
I20260808 02:12:01.699764 1024 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:01.699858 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:01.700109 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:01.702294 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "2287d09dfab546fa9c4d85e85fb2d4f8"
format_stamp: "Formatted at 2026-08-08 02:12:01 on dist-test-slave-bw88"
server_key: "f6172ba0a31145eadddc1c56ed6f2996"
server_key_iv: "dc61ad2793cd7a407328113f8f61de65"
server_key_version: "encryptionkey@0"
I20260808 02:12:01.702885 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "2287d09dfab546fa9c4d85e85fb2d4f8"
format_stamp: "Formatted at 2026-08-08 02:12:01 on dist-test-slave-bw88"
server_key: "f6172ba0a31145eadddc1c56ed6f2996"
server_key_iv: "dc61ad2793cd7a407328113f8f61de65"
server_key_version: "encryptionkey@0"
I20260808 02:12:01.706845 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:12:01.709491 1051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:01.710714 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:12:01.710861 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "2287d09dfab546fa9c4d85e85fb2d4f8"
format_stamp: "Formatted at 2026-08-08 02:12:01 on dist-test-slave-bw88"
server_key: "f6172ba0a31145eadddc1c56ed6f2996"
server_key_iv: "dc61ad2793cd7a407328113f8f61de65"
server_key_version: "encryptionkey@0"
I20260808 02:12:01.710979 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:01.720289 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:01.723191 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:01.723389 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:01.730760 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41131
I20260808 02:12:01.730789 1121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41131 every 8 connection(s)
I20260808 02:12:01.731688 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:01.734517 1123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:01.739975 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1024
I20260808 02:12:01.740134 1123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8: Bootstrap starting.
I20260808 02:12:01.740162 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:01.740479 5957 external_mini_cluster.cc:1428] Setting key dc3d018a893b6fc0f7f6367cc74503bc
I20260808 02:12:01.742811 1123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:01.743580 1123 log.cc:824] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:01.745761 1123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8: No bootstrap required, opened a new log
I20260808 02:12:01.749804 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:01.741912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46449 (local address 127.0.0.1:41131)
0808 02:12:01.742362 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:01.742378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:01.742417 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:12:01.743142 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:01.743162 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:01.743507 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:01.743862 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:01.744297 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:01.745325 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:01.746437 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:01.746920 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:01.747107 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:01.747977 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:01.747982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:01.747996 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:01.748389 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:01.748446 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:01.748468 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:01.748582 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:01.748788 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 02:12:01.749001 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
I20260808 02:12:01.749673 1123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260808 02:12:01.749909 1123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:01.749971 1123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2287d09dfab546fa9c4d85e85fb2d4f8, State: Initialized, Role: FOLLOWER
I20260808 02:12:01.750481 1123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260808 02:12:01.750656 1123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:01.750712 1123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:01.750806 1123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:01.751758 1123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260808 02:12:01.752091 1123 leader_election.cc:302] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2287d09dfab546fa9c4d85e85fb2d4f8; no voters:
I20260808 02:12:01.752429 1123 leader_election.cc:288] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:01.752540 1132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:01.752779 1132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [term 1 LEADER]: Becoming Leader. State: Replica: 2287d09dfab546fa9c4d85e85fb2d4f8, State: Running, Role: LEADER
I20260808 02:12:01.752871 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:01.753150 1132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } }
I20260808 02:12:01.753786 1123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:01.755293 1133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260808 02:12:01.755419 1133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:01.755700 1134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2287d09dfab546fa9c4d85e85fb2d4f8. Latest consensus state: current_term: 1 leader_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2287d09dfab546fa9c4d85e85fb2d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41131 } } }
I20260808 02:12:01.755800 1134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:01.755820 1149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:01.759262 1149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:01.760329 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:01.755340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34887 (local address 127.0.0.1:41131)
0808 02:12:01.755475 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:01.755480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:01.755766 (+ 286us) server_negotiation.cc:407] Connection header received
0808 02:12:01.755976 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:01.755979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:01.756025 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:01.756072 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:01.756490 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:01.757216 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:01.758388 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:01.758713 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:01.758788 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:01.759780 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:01.759784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:01.759788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:01.759802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:01.759830 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:01.759843 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:01.759889 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:01.760148 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:12:01.760202 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:12:01.765363 1149 catalog_manager.cc:1347] Generated new cluster ID: ed8024283fdd45dd9fd01a06441ac5cd
I20260808 02:12:01.765441 1149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:01.778788 1149 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:01.780160 1149 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:01.793497 1149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2287d09dfab546fa9c4d85e85fb2d4f8: Generated new TSK 0
I20260808 02:12:01.794245 1149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:01.804976 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1024
2026-08-08T02:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1591 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:01 dist-test-slave-bw88 krb5kdc[1166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:01 dist-test-slave-bw88 krb5kdc[1166](info): set up 2 sockets
Aug 08 02:12:01 dist-test-slave-bw88 krb5kdc[1166](info): commencing operation
krb5kdc: starting...
W20260808 02:12:03.023450 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:03 dist-test-slave-bw88 krb5kdc[1166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:03.167843 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:03.287397 1180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:03.287690 1180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:03.287734 1180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:03.291810 1180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:03.291875 1180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:03.291895 1180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:03.291913 1180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:03.291927 1180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:03.297499 1180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:03.298707 1180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:03.299872 1180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:03.309194 1186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:03.309194 1189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:03.309237 1185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:03.309248 1180 server_base.cc:1034] running on GCE node
I20260808 02:12:03.310111 1180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:03.311270 1180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:03.312507 1180 hybrid_clock.cc:648] HybridClock initialized: now 1786155123312493 us; error 27 us; skew 500 ppm
Aug 08 02:12:03 dist-test-slave-bw88 krb5kdc[1166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:03.315783 1180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:03.316799 1180 webserver.cc:460] Webserver started at http://127.0.0.1:36595/ using document root <none> and password file <none>
I20260808 02:12:03.317512 1180 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:03.317618 1180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:03.317889 1180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:03.320220 1180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "625075648370468cbc385af05c7bb08a"
format_stamp: "Formatted at 2026-08-08 02:12:03 on dist-test-slave-bw88"
server_key: "bf91a5e83ca10467939652ef90b6e4b3"
server_key_iv: "ae0bbcd4b56675ee291d3b8166518ba7"
server_key_version: "encryptionkey@0"
I20260808 02:12:03.320907 1180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "625075648370468cbc385af05c7bb08a"
format_stamp: "Formatted at 2026-08-08 02:12:03 on dist-test-slave-bw88"
server_key: "bf91a5e83ca10467939652ef90b6e4b3"
server_key_iv: "ae0bbcd4b56675ee291d3b8166518ba7"
server_key_version: "encryptionkey@0"
I20260808 02:12:03.325146 1180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 02:12:03.327973 1196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:03.329315 1180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 02:12:03.329612 1180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "625075648370468cbc385af05c7bb08a"
format_stamp: "Formatted at 2026-08-08 02:12:03 on dist-test-slave-bw88"
server_key: "bf91a5e83ca10467939652ef90b6e4b3"
server_key_iv: "ae0bbcd4b56675ee291d3b8166518ba7"
server_key_version: "encryptionkey@0"
I20260808 02:12:03.329743 1180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:03.334872 1180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:03.338358 1180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:03.338587 1180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:03.346802 1180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38855
I20260808 02:12:03.346832 1264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38855 every 8 connection(s)
I20260808 02:12:03.347838 1180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:03.350984 1265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:03.356081 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1180
I20260808 02:12:03.356261 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:03.356547 5957 external_mini_cluster.cc:1428] Setting key 95bb8fc2168b2e4db9bc78c5ba9cce99
I20260808 02:12:03.356917 1265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a: Bootstrap starting.
I20260808 02:12:03.359488 1265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:03.360469 1265 log.cc:824] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:03.362159 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:03.358005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38381 (local address 127.0.0.1:38855)
0808 02:12:03.358474 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:03.358489 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:03.358527 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:03.359171 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:03.359188 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:03.359478 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:03.359832 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:03.360771 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:03.360776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:03.360788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:03.360800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:03.360890 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:03.360933 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:03.361062 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:03.361186 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 02:12:03.361378 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20260808 02:12:03.362854 1265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a: No bootstrap required, opened a new log
I20260808 02:12:03.365252 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:03.367194 1265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260808 02:12:03.367517 1265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:03.367589 1265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625075648370468cbc385af05c7bb08a, State: Initialized, Role: FOLLOWER
I20260808 02:12:03.368106 1265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260808 02:12:03.368300 1265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:03.368376 1265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:03.368464 1265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:03.369320 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:03.367430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45405 (local address 127.0.0.1:38855)
0808 02:12:03.367548 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:03.367554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:03.367781 (+ 227us) server_negotiation.cc:407] Connection header received
0808 02:12:03.367906 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:03.367909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:03.367961 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:03.368029 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:03.368914 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:03.368917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:03.368919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:03.368930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:03.368955 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:03.368966 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:03.369044 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:03.369148 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 02:12:03.369178 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:12:03.369779 1265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260808 02:12:03.370163 1265 leader_election.cc:302] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625075648370468cbc385af05c7bb08a; no voters:
I20260808 02:12:03.370535 1265 leader_election.cc:288] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:03.370800 1277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:03.371032 1277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [term 1 LEADER]: Becoming Leader. State: Replica: 625075648370468cbc385af05c7bb08a, State: Running, Role: LEADER
I20260808 02:12:03.371371 1277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260808 02:12:03.371856 1265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:03.373193 1278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625075648370468cbc385af05c7bb08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260808 02:12:03.373214 1279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625075648370468cbc385af05c7bb08a. Latest consensus state: current_term: 1 leader_uuid: "625075648370468cbc385af05c7bb08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625075648370468cbc385af05c7bb08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260808 02:12:03.373373 1278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:03.373402 1279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:03.373797 1291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:03.377197 1291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:03.383170 1291 catalog_manager.cc:1347] Generated new cluster ID: dfc7c7bf28404ab4af09a79ff322c557
I20260808 02:12:03.383241 1291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:03.402261 1291 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:03.402661 1291 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:03.409984 1291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625075648370468cbc385af05c7bb08a: Generated new TSK 0
I20260808 02:12:03.410847 1291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:03.426409 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1180
2026-08-08T02:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1616 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:03 dist-test-slave-bw88 krb5kdc[1306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:03 dist-test-slave-bw88 krb5kdc[1306](info): set up 2 sockets
Aug 08 02:12:03 dist-test-slave-bw88 krb5kdc[1306](info): commencing operation
krb5kdc: starting...
W20260808 02:12:04.618536 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:04 dist-test-slave-bw88 krb5kdc[1306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:04.748661 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:04.866732 1324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:04.867039 1324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:04.867112 1324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:04.871433 1324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:04.871521 1324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:04.871568 1324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:04.871605 1324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:04.871640 1324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:04.877264 1324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:04.878532 1324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:04.879680 1324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:04.888862 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:04.888974 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:04.888870 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:04.889233 1324 server_base.cc:1034] running on GCE node
I20260808 02:12:04.889894 1324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:04.891077 1324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:04.892253 1324 hybrid_clock.cc:648] HybridClock initialized: now 1786155124892233 us; error 33 us; skew 500 ppm
Aug 08 02:12:04 dist-test-slave-bw88 krb5kdc[1306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:04.901602 1324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:04.902711 1324 webserver.cc:460] Webserver started at http://127.0.0.1:39105/ using document root <none> and password file <none>
I20260808 02:12:04.903352 1324 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:04.903446 1324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:04.903702 1324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:04.905994 1324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "a05346407632493daf06f315938175f5"
format_stamp: "Formatted at 2026-08-08 02:12:04 on dist-test-slave-bw88"
server_key: "2a8a44e621116e58b02a4483b5fe087a"
server_key_iv: "773fc3614dd73420a8f7040215417ac7"
server_key_version: "encryptionkey@0"
I20260808 02:12:04.906605 1324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "a05346407632493daf06f315938175f5"
format_stamp: "Formatted at 2026-08-08 02:12:04 on dist-test-slave-bw88"
server_key: "2a8a44e621116e58b02a4483b5fe087a"
server_key_iv: "773fc3614dd73420a8f7040215417ac7"
server_key_version: "encryptionkey@0"
I20260808 02:12:04.910596 1324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:12:04.913305 1342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:04.914871 1324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:04.915052 1324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "a05346407632493daf06f315938175f5"
format_stamp: "Formatted at 2026-08-08 02:12:04 on dist-test-slave-bw88"
server_key: "2a8a44e621116e58b02a4483b5fe087a"
server_key_iv: "773fc3614dd73420a8f7040215417ac7"
server_key_version: "encryptionkey@0"
I20260808 02:12:04.915189 1324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:04.934340 1324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:04.937929 1324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:04.938158 1324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:04.946040 1324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41049
I20260808 02:12:04.946099 1406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41049 every 8 connection(s)
I20260808 02:12:04.947104 1324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:04.950426 1408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:04.956219 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1324
I20260808 02:12:04.956389 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:04.956745 5957 external_mini_cluster.cc:1428] Setting key 00a06ecc0b3b44729a006ea99fd42250
I20260808 02:12:04.957304 1408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5: Bootstrap starting.
I20260808 02:12:04.960130 1408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:04.960994 1408 log.cc:824] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:04.963326 1408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5: No bootstrap required, opened a new log
Aug 08 02:12:04 dist-test-slave-bw88 krb5kdc[1306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:12:04.968003 1408 raft_consensus.cc:348] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } }
I20260808 02:12:04.968246 1408 raft_consensus.cc:374] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:04.968329 1408 raft_consensus.cc:729] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05346407632493daf06f315938175f5, State: Initialized, Role: FOLLOWER
I20260808 02:12:04.968868 1408 consensus_queue.cc:260] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } }
I20260808 02:12:04.969051 1408 raft_consensus.cc:388] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:04.969152 1408 raft_consensus.cc:482] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:04.969286 1408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:04.970289 1408 raft_consensus.cc:504] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } }
I20260808 02:12:04.970687 1408 leader_election.cc:302] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a05346407632493daf06f315938175f5; no voters:
I20260808 02:12:04.971100 1408 leader_election.cc:288] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:04.971195 1416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:04.971457 1416 raft_consensus.cc:686] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [term 1 LEADER]: Becoming Leader. State: Replica: a05346407632493daf06f315938175f5, State: Running, Role: LEADER
I20260808 02:12:04.971772 1416 consensus_queue.cc:237] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } }
I20260808 02:12:04.972201 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:04.958366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:38949 (local address 127.0.0.1:41049)
0808 02:12:04.958882 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:04.958898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:04.958937 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:12:04.959601 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:04.959619 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:04.959975 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:04.960328 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:04.961207 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:04.962282 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:04.963157 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:04.963693 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:04.963889 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:04.966855 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:04.966860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:04.966873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:04.966900 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:04.968813 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:04.969435 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:04.969440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:04.969446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:04.969563 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:04.969916 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:04.969919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:04.969923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:04.970323 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:04.970570 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:04.970885 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:12:04.971183 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":165,"threads_started":1}
I20260808 02:12:04.972654 1408 sys_catalog.cc:564] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:04.973040 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a05346407632493daf06f315938175f5. Latest consensus state: current_term: 1 leader_uuid: "a05346407632493daf06f315938175f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } } }
I20260808 02:12:04.973162 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:04.973605 1419 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a05346407632493daf06f315938175f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05346407632493daf06f315938175f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41049 } } }
I20260808 02:12:04.973726 1419 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:04.973771 1424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:04.975600 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:04.977222 1424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:04.983179 1424 catalog_manager.cc:1347] Generated new cluster ID: a271f2be6e664b919adb25bcbbe7c043
I20260808 02:12:04.983253 1424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:04.988765 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:04.977959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:59339 (local address 127.0.0.1:41049)
0808 02:12:04.978085 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:04.978091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:04.978358 (+ 267us) server_negotiation.cc:407] Connection header received
0808 02:12:04.978682 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:04.978686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:04.978745 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:04.978848 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:04.979905 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:04.980660 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:04.982077 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:04.982529 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:04.982617 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:04.984913 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:04.984917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:04.984920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:04.984944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:04.986724 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:04.987348 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:04.987370 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:04.987372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:04.987453 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:04.987840 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:04.987842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:04.987844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:04.988032 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:04.988174 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:04.988510 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 02:12:04.988607 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:12:04.997990 1424 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:04.999430 1424 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:05.009366 1424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a05346407632493daf06f315938175f5: Generated new TSK 0
I20260808 02:12:05.010288 1424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:05.014282 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1324
W20260808 02:12:05.030467 1303 connection.cc:535] client connection to 127.0.0.1:41049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41049): Connection reset by peer (error 104)
2026-08-08T02:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1591 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:05 dist-test-slave-bw88 krb5kdc[1457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:05 dist-test-slave-bw88 krb5kdc[1457](info): set up 2 sockets
Aug 08 02:12:05 dist-test-slave-bw88 krb5kdc[1457](info): commencing operation
krb5kdc: starting...
W20260808 02:12:06.210112 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:06 dist-test-slave-bw88 krb5kdc[1457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:06.344115 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:06.460268 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:06.460597 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:06.460641 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:06.464690 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:06.464747 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:06.464766 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:06.464783 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:06.464798 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:06.469946 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:06.471104 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:06.472198 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:06.481032 1479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:06.481169 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:06.481032 1480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:06.481293 1472 server_base.cc:1034] running on GCE node
I20260808 02:12:06.481984 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:06.483142 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:06.484336 1472 hybrid_clock.cc:648] HybridClock initialized: now 1786155126484307 us; error 39 us; skew 500 ppm
Aug 08 02:12:06 dist-test-slave-bw88 krb5kdc[1457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:06.487120 1472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:06.488066 1472 webserver.cc:460] Webserver started at http://127.0.0.1:41459/ using document root <none> and password file <none>
I20260808 02:12:06.488639 1472 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:06.488734 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:06.488981 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:06.491170 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "351d55b658234a65a60b5cc318034ba0"
format_stamp: "Formatted at 2026-08-08 02:12:06 on dist-test-slave-bw88"
server_key: "67ae0956b2fb83709ea1e80d14e70937"
server_key_iv: "b381d5d45222d195b10dea685fc8ce51"
server_key_version: "encryptionkey@0"
I20260808 02:12:06.491771 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "351d55b658234a65a60b5cc318034ba0"
format_stamp: "Formatted at 2026-08-08 02:12:06 on dist-test-slave-bw88"
server_key: "67ae0956b2fb83709ea1e80d14e70937"
server_key_iv: "b381d5d45222d195b10dea685fc8ce51"
server_key_version: "encryptionkey@0"
I20260808 02:12:06.495667 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:06.498175 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:06.499421 1472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:12:06.499569 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "351d55b658234a65a60b5cc318034ba0"
format_stamp: "Formatted at 2026-08-08 02:12:06 on dist-test-slave-bw88"
server_key: "67ae0956b2fb83709ea1e80d14e70937"
server_key_iv: "b381d5d45222d195b10dea685fc8ce51"
server_key_version: "encryptionkey@0"
I20260808 02:12:06.499691 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:06.531021 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:06.534322 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:06.534541 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:06.542348 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39881
I20260808 02:12:06.542372 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39881 every 8 connection(s)
I20260808 02:12:06.543416 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:06.546373 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:06.552102 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1472
I20260808 02:12:06.552244 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0: Bootstrap starting.
I20260808 02:12:06.552289 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:06.552685 5957 external_mini_cluster.cc:1428] Setting key 4d84237c98d1a95ab48bc2273ecd231d
I20260808 02:12:06.555012 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:06.555806 1556 log.cc:824] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:06.558086 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0: No bootstrap required, opened a new log
I20260808 02:12:06.562021 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } }
I20260808 02:12:06.562278 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:06.562356 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351d55b658234a65a60b5cc318034ba0, State: Initialized, Role: FOLLOWER
I20260808 02:12:06.562399 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:06.554185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:36893 (local address 127.0.0.1:39881)
0808 02:12:06.554733 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:06.554750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:06.554790 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:12:06.555527 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:06.555549 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:06.555914 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:06.556315 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:06.556766 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:06.557931 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:06.558913 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:06.559404 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:06.559635 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:06.560599 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:06.560605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:06.560618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:06.561005 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:06.561060 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:06.561083 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:06.561243 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:06.561457 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 02:12:06.561615 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":146,"threads_started":1}
I20260808 02:12:06.562888 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } }
I20260808 02:12:06.563069 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:06.563138 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:06.563233 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:06.564163 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } }
I20260808 02:12:06.564548 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351d55b658234a65a60b5cc318034ba0; no voters:
I20260808 02:12:06.564952 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:06.565068 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:06.565357 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [term 1 LEADER]: Becoming Leader. State: Replica: 351d55b658234a65a60b5cc318034ba0, State: Running, Role: LEADER
I20260808 02:12:06.565738 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } }
I20260808 02:12:06.565886 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:06.566365 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:06.568466 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351d55b658234a65a60b5cc318034ba0. Latest consensus state: current_term: 1 leader_uuid: "351d55b658234a65a60b5cc318034ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } } }
I20260808 02:12:06.568607 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:06.568992 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351d55b658234a65a60b5cc318034ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d55b658234a65a60b5cc318034ba0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39881 } } }
I20260808 02:12:06.569437 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:06.569482 1588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:06.572611 1588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:06.573755 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:06.568235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:46595 (local address 127.0.0.1:39881)
0808 02:12:06.568327 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:06.568331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:06.569395 (+ 1064us) server_negotiation.cc:407] Connection header received
0808 02:12:06.569569 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:06.569572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:06.569624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:06.569712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:06.570006 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:06.570756 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:06.571895 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:06.572226 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:06.572334 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:06.573205 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:06.573209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:06.573213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:06.573227 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:06.573255 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:06.573269 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:06.573401 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:06.573590 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 02:12:06.573628 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 02:12:06.578828 1588 catalog_manager.cc:1347] Generated new cluster ID: 540176cf220f48959fc1167e68055f9e
I20260808 02:12:06.578914 1588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:06.596277 1588 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:06.597609 1588 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:06.602330 1588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 351d55b658234a65a60b5cc318034ba0: Generated new TSK 0
I20260808 02:12:06.603117 1588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:06.610558 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1472
W20260808 02:12:06.623453 1455 connection.cc:535] client connection to 127.0.0.1:39881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39881): Connection reset by peer (error 104)
2026-08-08T02:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:06 dist-test-slave-bw88 krb5kdc[1600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:06 dist-test-slave-bw88 krb5kdc[1600](info): set up 2 sockets
Aug 08 02:12:06 dist-test-slave-bw88 krb5kdc[1600](info): commencing operation
krb5kdc: starting...
W20260808 02:12:07.858217 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:07 dist-test-slave-bw88 krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:07.994194 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:08.114228 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:08.114552 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:08.114640 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:08.118700 1619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:08.118791 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:08.118844 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:08.118901 1619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:08.118947 1619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:08.124313 1619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:08.125432 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:08.126565 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:08.135613 1632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:08.135636 1627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:08.135687 1619 server_base.cc:1034] running on GCE node
W20260808 02:12:08.135741 1626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:08.136430 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:08.137672 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:08.138849 1619 hybrid_clock.cc:648] HybridClock initialized: now 1786155128138817 us; error 43 us; skew 500 ppm
Aug 08 02:12:08 dist-test-slave-bw88 krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:08.141834 1619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:08.142661 1619 webserver.cc:460] Webserver started at http://127.0.0.1:43477/ using document root <none> and password file <none>
I20260808 02:12:08.143191 1619 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:08.143247 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:08.143412 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:08.145475 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "9fc57f88af0b4680be34fbaf7eb67236"
format_stamp: "Formatted at 2026-08-08 02:12:08 on dist-test-slave-bw88"
server_key: "6d03c218d52b44637ea5fa9ea0ba838a"
server_key_iv: "d486eaf55b25f4fba48125f046783661"
server_key_version: "encryptionkey@0"
I20260808 02:12:08.146013 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "9fc57f88af0b4680be34fbaf7eb67236"
format_stamp: "Formatted at 2026-08-08 02:12:08 on dist-test-slave-bw88"
server_key: "6d03c218d52b44637ea5fa9ea0ba838a"
server_key_iv: "d486eaf55b25f4fba48125f046783661"
server_key_version: "encryptionkey@0"
I20260808 02:12:08.149765 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:12:08.152518 1639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:08.153877 1619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:08.154023 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9fc57f88af0b4680be34fbaf7eb67236"
format_stamp: "Formatted at 2026-08-08 02:12:08 on dist-test-slave-bw88"
server_key: "6d03c218d52b44637ea5fa9ea0ba838a"
server_key_iv: "d486eaf55b25f4fba48125f046783661"
server_key_version: "encryptionkey@0"
I20260808 02:12:08.154142 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:08.158787 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:08.162045 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:08.162240 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:08.169472 1619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34941
I20260808 02:12:08.169775 1703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34941 every 8 connection(s)
I20260808 02:12:08.170434 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:08.171325 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1619
I20260808 02:12:08.171478 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:08.171833 5957 external_mini_cluster.cc:1428] Setting key 4729e832ff016e49548fd0b48a90a9a0
I20260808 02:12:08.173990 1704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:08.177209 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:08.173321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:45157 (local address 127.0.0.1:34941)
0808 02:12:08.173830 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:08.173853 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:08.173888 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:12:08.174583 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:08.174601 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:08.174882 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:08.175244 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:08.176044 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:08.176054 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:08.176066 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:08.176077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:08.176129 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:08.176148 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:08.176299 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:08.176369 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 02:12:08.176531 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":133,"threads_started":1}
I20260808 02:12:08.180038 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:08.180291 1704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236: Bootstrap starting.
I20260808 02:12:08.182955 1704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:08.183928 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:08.182172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:39527 (local address 127.0.0.1:34941)
0808 02:12:08.182301 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:08.182306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:08.182566 (+ 260us) server_negotiation.cc:407] Connection header received
0808 02:12:08.182681 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:08.182683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:08.182715 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:08.182779 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:08.183519 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:08.183522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:08.183524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:08.183534 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:08.183559 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:08.183570 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:08.183637 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:08.183764 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:12:08.183796 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:12:08.183928 1704 log.cc:824] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:08.186210 1704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236: No bootstrap required, opened a new log
I20260808 02:12:08.189759 1704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260808 02:12:08.189978 1704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:08.190059 1704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc57f88af0b4680be34fbaf7eb67236, State: Initialized, Role: FOLLOWER
I20260808 02:12:08.190551 1704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260808 02:12:08.190732 1704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:08.190812 1704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:08.190919 1704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:08.191776 1704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260808 02:12:08.192077 1704 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc57f88af0b4680be34fbaf7eb67236; no voters:
I20260808 02:12:08.192332 1704 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:08.192484 1719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:08.192742 1719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc57f88af0b4680be34fbaf7eb67236, State: Running, Role: LEADER
I20260808 02:12:08.193070 1719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } }
I20260808 02:12:08.193626 1704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:08.194804 1721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc57f88af0b4680be34fbaf7eb67236. Latest consensus state: current_term: 1 leader_uuid: "9fc57f88af0b4680be34fbaf7eb67236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260808 02:12:08.194949 1721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:08.195261 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc57f88af0b4680be34fbaf7eb67236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc57f88af0b4680be34fbaf7eb67236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34941 } } }
I20260808 02:12:08.195348 1729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:08.195446 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:08.198581 1729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:08.204387 1729 catalog_manager.cc:1347] Generated new cluster ID: 4de3edac64c64903b7bbff6756fda7c6
I20260808 02:12:08.204530 1729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:08.224069 1729 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:08.224416 1729 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:08.239739 1729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc57f88af0b4680be34fbaf7eb67236: Generated new TSK 0
I20260808 02:12:08.240608 1729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:08.252151 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1619
2026-08-08T02:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1641 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:08 dist-test-slave-bw88 krb5kdc[1750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:08 dist-test-slave-bw88 krb5kdc[1750](info): set up 2 sockets
Aug 08 02:12:08 dist-test-slave-bw88 krb5kdc[1750](info): commencing operation
krb5kdc: starting...
W20260808 02:12:09.441890 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:09.572813 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:09.690341 1767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:09.690663 1767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:09.690713 1767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:09.695282 1767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:09.695354 1767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:09.695375 1767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:09.695392 1767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:09.695407 1767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:09.700822 1767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:09.701932 1767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:09.702950 1767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:12:09.711819 1767 server_base.cc:1034] running on GCE node
W20260808 02:12:09.711792 1773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:09.711804 1779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:09.712013 1775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:09.712745 1767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:09.713933 1767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:09.715126 1767 hybrid_clock.cc:648] HybridClock initialized: now 1786155129715091 us; error 48 us; skew 500 ppm
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:09.718010 1767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:09.718976 1767 webserver.cc:460] Webserver started at http://127.0.0.1:41603/ using document root <none> and password file <none>
I20260808 02:12:09.719566 1767 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:09.719662 1767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:09.719910 1767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:09.722123 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "04c3600ac4b5449482407e5d7d83beac"
format_stamp: "Formatted at 2026-08-08 02:12:09 on dist-test-slave-bw88"
server_key: "e4c1cd0706b6cdc08928b7e723f3b746"
server_key_iv: "a5e1edc18fbc5b71ef72cdf18312264d"
server_key_version: "encryptionkey@0"
I20260808 02:12:09.722718 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "04c3600ac4b5449482407e5d7d83beac"
format_stamp: "Formatted at 2026-08-08 02:12:09 on dist-test-slave-bw88"
server_key: "e4c1cd0706b6cdc08928b7e723f3b746"
server_key_iv: "a5e1edc18fbc5b71ef72cdf18312264d"
server_key_version: "encryptionkey@0"
I20260808 02:12:09.726572 1767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:12:09.729173 1789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:09.730494 1767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:12:09.730674 1767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "04c3600ac4b5449482407e5d7d83beac"
format_stamp: "Formatted at 2026-08-08 02:12:09 on dist-test-slave-bw88"
server_key: "e4c1cd0706b6cdc08928b7e723f3b746"
server_key_iv: "a5e1edc18fbc5b71ef72cdf18312264d"
server_key_version: "encryptionkey@0"
I20260808 02:12:09.730790 1767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:09.744414 1767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:09.747444 1767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:09.747689 1767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:09.755035 1767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43421
I20260808 02:12:09.755041 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43421 every 8 connection(s)
I20260808 02:12:09.756011 1767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:09.758939 1856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:09.760159 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1767
I20260808 02:12:09.760324 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:09.760633 5957 external_mini_cluster.cc:1428] Setting key ceebe72d2c9ce7eaa3029dcd09d99d6c
I20260808 02:12:09.765486 1856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac: Bootstrap starting.
I20260808 02:12:09.768213 1856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:09.769181 1856 log.cc:824] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:12:09.771462 1856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac: No bootstrap required, opened a new log
I20260808 02:12:09.775372 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:09.762041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:49167 (local address 127.0.0.1:43421)
0808 02:12:09.762440 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:09.762467 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:09.762504 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:12:09.763140 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:09.763157 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:09.763497 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:09.763770 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:09.764787 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:09.765893 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:09.767066 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:09.767530 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:09.767646 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:09.770634 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:09.770639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:09.770653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:09.770678 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:09.772382 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:09.772934 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:09.772938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:09.772943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:09.773064 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:09.773440 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:09.773443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:09.773444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:09.773826 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:09.773997 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:09.774346 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 02:12:09.774612 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":134,"threads_started":1}
I20260808 02:12:09.775698 1856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260808 02:12:09.775979 1856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:09.776042 1856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c3600ac4b5449482407e5d7d83beac, State: Initialized, Role: FOLLOWER
I20260808 02:12:09.776521 1856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260808 02:12:09.776695 1856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:09.776785 1856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:09.776906 1856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:09.778062 1856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260808 02:12:09.778437 1856 leader_election.cc:302] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c3600ac4b5449482407e5d7d83beac; no voters:
I20260808 02:12:09.778738 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:09.778787 1856 leader_election.cc:288] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:09.778893 1865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:09.779136 1865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [term 1 LEADER]: Becoming Leader. State: Replica: 04c3600ac4b5449482407e5d7d83beac, State: Running, Role: LEADER
I20260808 02:12:09.779536 1865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260808 02:12:09.780222 1856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:09.780865 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c3600ac4b5449482407e5d7d83beac. Latest consensus state: current_term: 1 leader_uuid: "04c3600ac4b5449482407e5d7d83beac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } } }
I20260808 02:12:09.781060 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:09.781412 1866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c3600ac4b5449482407e5d7d83beac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c3600ac4b5449482407e5d7d83beac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } } }
I20260808 02:12:09.781513 1866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:09.781580 1876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:09.785071 1876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:09.791534 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:09.781085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:39831 (local address 127.0.0.1:43421)
0808 02:12:09.781218 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:09.781223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:09.781860 (+ 637us) server_negotiation.cc:407] Connection header received
0808 02:12:09.782049 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:09.782053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:09.782101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:09.782150 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:09.783141 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:09.783875 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:09.785685 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:09.786008 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:09.786098 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:09.787955 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:09.787966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:09.787968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:09.787990 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:09.789590 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:09.790176 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:09.790181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:09.790183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:09.790278 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:09.790641 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:09.790645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:09.790647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:09.790829 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:09.790988 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:09.791272 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:12:09.791364 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 02:12:09.791692 1876 catalog_manager.cc:1347] Generated new cluster ID: 53cfc258ef534baaa4dd1867034724f9
I20260808 02:12:09.791738 1876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:09.802203 1876 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:09.803402 1876 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:09.810042 1876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c3600ac4b5449482407e5d7d83beac: Generated new TSK 0
I20260808 02:12:09.810705 1876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:09.819885 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1767
W20260808 02:12:09.835899 1745 connection.cc:535] client connection to 127.0.0.1:43421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43421): Connection reset by peer (error 104)
2026-08-08T02:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1571 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1893](info): set up 2 sockets
Aug 08 02:12:09 dist-test-slave-bw88 krb5kdc[1893](info): commencing operation
krb5kdc: starting...
W20260808 02:12:11.038036 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:11 dist-test-slave-bw88 krb5kdc[1893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:11.174194 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:11.291234 1908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:11.291556 1908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:11.291637 1908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:11.295601 1908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:11.295702 1908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:11.295742 1908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:11.295800 1908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:11.295842 1908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:11.301052 1908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:11.302191 1908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:11.303244 1908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:11.312191 1919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:11.312232 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:11.312350 1908 server_base.cc:1034] running on GCE node
W20260808 02:12:11.312232 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:11.312943 1908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:11.314127 1908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:11.315331 1908 hybrid_clock.cc:648] HybridClock initialized: now 1786155131315310 us; error 32 us; skew 500 ppm
Aug 08 02:12:11 dist-test-slave-bw88 krb5kdc[1893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:11.321687 1908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:11.322666 1908 webserver.cc:460] Webserver started at http://127.0.0.1:37531/ using document root <none> and password file <none>
I20260808 02:12:11.323253 1908 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:11.323347 1908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:11.323606 1908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:11.325762 1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "c13b225a0c9d416dba7cb1b968ae962f"
format_stamp: "Formatted at 2026-08-08 02:12:11 on dist-test-slave-bw88"
server_key: "7d419930b26533ad2da23acda716760a"
server_key_iv: "d875eb9668e37d92e067ebb63295e24a"
server_key_version: "encryptionkey@0"
I20260808 02:12:11.326350 1908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "c13b225a0c9d416dba7cb1b968ae962f"
format_stamp: "Formatted at 2026-08-08 02:12:11 on dist-test-slave-bw88"
server_key: "7d419930b26533ad2da23acda716760a"
server_key_iv: "d875eb9668e37d92e067ebb63295e24a"
server_key_version: "encryptionkey@0"
I20260808 02:12:11.330082 1908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:12:11.332564 1929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:11.333878 1908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:12:11.334043 1908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "c13b225a0c9d416dba7cb1b968ae962f"
format_stamp: "Formatted at 2026-08-08 02:12:11 on dist-test-slave-bw88"
server_key: "7d419930b26533ad2da23acda716760a"
server_key_iv: "d875eb9668e37d92e067ebb63295e24a"
server_key_version: "encryptionkey@0"
I20260808 02:12:11.334159 1908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:11.352921 1908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:11.356009 1908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:11.356220 1908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:11.363976 1908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39615
I20260808 02:12:11.364007 2004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39615 every 8 connection(s)
I20260808 02:12:11.364936 1908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:11.367856 2006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:11.371935 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 1908
I20260808 02:12:11.372115 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:11.372438 5957 external_mini_cluster.cc:1428] Setting key 576bb31a984f1987078810e78d3c5c20
I20260808 02:12:11.374217 2006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f: Bootstrap starting.
I20260808 02:12:11.376565 2006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:11.377324 2006 log.cc:824] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:11.379419 2006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f: No bootstrap required, opened a new log
I20260808 02:12:11.382306 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:11.374024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:41043 (local address 127.0.0.1:39615)
0808 02:12:11.374573 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:11.374589 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:11.374627 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:11.375293 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:11.375309 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:11.375659 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:11.376010 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:11.376428 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:11.377582 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:11.378495 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:11.378982 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:11.379154 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:11.380128 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:11.380133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:11.380147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:11.380542 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:11.380598 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:11.380622 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:11.380768 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:11.380961 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 02:12:11.381115 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":158,"threads_started":1}
I20260808 02:12:11.382761 2006 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260808 02:12:11.383011 2006 raft_consensus.cc:374] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:11.383105 2006 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13b225a0c9d416dba7cb1b968ae962f, State: Initialized, Role: FOLLOWER
I20260808 02:12:11.383615 2006 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260808 02:12:11.383785 2006 raft_consensus.cc:388] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:11.383849 2006 raft_consensus.cc:482] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:11.383945 2006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:11.385099 2006 raft_consensus.cc:504] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260808 02:12:11.385494 2006 leader_election.cc:302] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13b225a0c9d416dba7cb1b968ae962f; no voters:
I20260808 02:12:11.385624 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:11.385910 2006 leader_election.cc:288] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:11.385995 2014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:11.386233 2014 raft_consensus.cc:686] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [term 1 LEADER]: Becoming Leader. State: Replica: c13b225a0c9d416dba7cb1b968ae962f, State: Running, Role: LEADER
I20260808 02:12:11.386652 2014 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260808 02:12:11.387248 2006 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:11.388224 2014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13b225a0c9d416dba7cb1b968ae962f. Latest consensus state: current_term: 1 leader_uuid: "c13b225a0c9d416dba7cb1b968ae962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } } }
I20260808 02:12:11.388384 2014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:11.388688 2015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13b225a0c9d416dba7cb1b968ae962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13b225a0c9d416dba7cb1b968ae962f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } } }
I20260808 02:12:11.388798 2015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:11.388782 2030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:11.392249 2030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:11.394353 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:11.387914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:51925 (local address 127.0.0.1:39615)
0808 02:12:11.388034 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:11.388038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:11.389613 (+ 1575us) server_negotiation.cc:407] Connection header received
0808 02:12:11.389806 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:11.389810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:11.389862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:11.389941 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:11.390231 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:11.390994 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:11.392098 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:11.392443 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:11.392552 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:11.393670 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:11.393674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:11.393676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:11.393690 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:11.393720 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:11.393733 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:11.393825 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:11.394189 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 02:12:11.394227 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:12:11.398435 2030 catalog_manager.cc:1347] Generated new cluster ID: 339a2dc261a64698b1a79e752b2014e2
I20260808 02:12:11.398531 2030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:11.412693 2030 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:11.413729 2030 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:11.419857 2030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c13b225a0c9d416dba7cb1b968ae962f: Generated new TSK 0
I20260808 02:12:11.420542 2030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:11.432245 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 1908
W20260808 02:12:11.446759 1889 connection.cc:535] client connection to 127.0.0.1:39615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39615): Connection reset by peer (error 104)
2026-08-08T02:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1610 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:11 dist-test-slave-bw88 krb5kdc[2044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:11 dist-test-slave-bw88 krb5kdc[2044](info): set up 2 sockets
Aug 08 02:12:11 dist-test-slave-bw88 krb5kdc[2044](info): commencing operation
krb5kdc: starting...
W20260808 02:12:12.640925 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:12 dist-test-slave-bw88 krb5kdc[2044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:12:12.774435 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:12.890938 2059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:12.891261 2059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:12.891355 2059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:12.895370 2059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:12.895465 2059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:12.895505 2059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:12.895548 2059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:12.895579 2059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:12.901091 2059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:12.902294 2059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:12.903378 2059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:12.912194 2069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:12.912164 2066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:12.912443 2059 server_base.cc:1034] running on GCE node
W20260808 02:12:12.912279 2067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:12.913183 2059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:12.914364 2059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:12.915546 2059 hybrid_clock.cc:648] HybridClock initialized: now 1786155132915510 us; error 45 us; skew 500 ppm
Aug 08 02:12:12 dist-test-slave-bw88 krb5kdc[2044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:12.918843 2059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:12:12.919762 2059 webserver.cc:460] Webserver started at http://127.0.0.1:46869/ using document root <none> and password file <none>
I20260808 02:12:12.920336 2059 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:12.920398 2059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:12.920658 2059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:12.922808 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7"
format_stamp: "Formatted at 2026-08-08 02:12:12 on dist-test-slave-bw88"
server_key: "2fd21e7c52acc4c3d010f7a1e2bf4813"
server_key_iv: "2ef3bee29d1b35b940ec4b677f8e998d"
server_key_version: "encryptionkey@0"
I20260808 02:12:12.923409 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7"
format_stamp: "Formatted at 2026-08-08 02:12:12 on dist-test-slave-bw88"
server_key: "2fd21e7c52acc4c3d010f7a1e2bf4813"
server_key_iv: "2ef3bee29d1b35b940ec4b677f8e998d"
server_key_version: "encryptionkey@0"
I20260808 02:12:12.927275 2059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:12.929718 2076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:12.930987 2059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:12:12.931150 2059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7"
format_stamp: "Formatted at 2026-08-08 02:12:12 on dist-test-slave-bw88"
server_key: "2fd21e7c52acc4c3d010f7a1e2bf4813"
server_key_iv: "2ef3bee29d1b35b940ec4b677f8e998d"
server_key_version: "encryptionkey@0"
I20260808 02:12:12.931253 2059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:12.935906 2059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:12.938870 2059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:12.939070 2059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:12.946877 2059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41813
I20260808 02:12:12.946902 2152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41813 every 8 connection(s)
I20260808 02:12:12.947824 2059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:12.950735 2153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:12.951450 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2059
I20260808 02:12:12.951613 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:12.951934 5957 external_mini_cluster.cc:1428] Setting key 05f834567886eee9fa3add8bc8956239
I20260808 02:12:12.957120 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:12.953369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:46101 (local address 127.0.0.1:41813)
0808 02:12:12.953806 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:12.953821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:12.953856 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:12:12.954510 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:12.954526 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:12.954812 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:12.955155 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:12.955981 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:12.955986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:12.955998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:12.956009 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:12.956063 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:12.956082 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:12.956230 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:12.956298 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:12:12.956461 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260808 02:12:12.957262 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7: Bootstrap starting.
I20260808 02:12:12.959916 2153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:12.960290 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:12.960799 2153 log.cc:824] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:12.963265 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7: No bootstrap required, opened a new log
I20260808 02:12:12.964155 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:12.962351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.189:58577 (local address 127.0.0.1:41813)
0808 02:12:12.962476 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:12.962482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:12.962650 (+ 168us) server_negotiation.cc:407] Connection header received
0808 02:12:12.962844 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:12.962846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:12.962878 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:12.962960 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:12.963665 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:12.963669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:12.963671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:12.963689 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:12.963714 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:12.963724 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:12.963795 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:12.963907 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 02:12:12.963936 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:12:12.967312 2153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260808 02:12:12.967599 2153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:12.967713 2153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fdcd2dd4864ca6ad9f8fbe3ddc54a7, State: Initialized, Role: FOLLOWER
I20260808 02:12:12.968179 2153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260808 02:12:12.968340 2153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:12.968432 2153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:12.968544 2153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:12.969487 2153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260808 02:12:12.969844 2153 leader_election.cc:302] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18fdcd2dd4864ca6ad9f8fbe3ddc54a7; no voters:
I20260808 02:12:12.970150 2153 leader_election.cc:288] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:12.970250 2165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:12.970513 2165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [term 1 LEADER]: Becoming Leader. State: Replica: 18fdcd2dd4864ca6ad9f8fbe3ddc54a7, State: Running, Role: LEADER
I20260808 02:12:12.970851 2165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260808 02:12:12.971496 2153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:12.972609 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fdcd2dd4864ca6ad9f8fbe3ddc54a7. Latest consensus state: current_term: 1 leader_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260808 02:12:12.972735 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:12.973152 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fdcd2dd4864ca6ad9f8fbe3ddc54a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260808 02:12:12.973799 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:12.973812 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:12.976471 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:12.981863 2179 catalog_manager.cc:1347] Generated new cluster ID: 269b278a547a4d53a79548c8eddcc36f
I20260808 02:12:12.981942 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:13.008921 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:13.009306 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:13.014547 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18fdcd2dd4864ca6ad9f8fbe3ddc54a7: Generated new TSK 0
I20260808 02:12:13.015273 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:13.020767 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2059
2026-08-08T02:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:13 dist-test-slave-bw88 krb5kdc[2195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:13 dist-test-slave-bw88 krb5kdc[2195](info): set up 2 sockets
Aug 08 02:12:13 dist-test-slave-bw88 krb5kdc[2195](info): commencing operation
krb5kdc: starting...
W20260808 02:12:14.234917 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:14.368181 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38283
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:14.484078 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:14.484380 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:14.484462 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:14.488397 2212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:14.488483 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:14.488521 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:14.488591 2212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:14.488631 2212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:14.493891 2212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:38283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:14.494982 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:14.496006 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:14.504834 2217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:14.504873 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:14.504938 2212 server_base.cc:1034] running on GCE node
W20260808 02:12:14.504884 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:14.505640 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:14.506842 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:14.508038 2212 hybrid_clock.cc:648] HybridClock initialized: now 1786155134508013 us; error 40 us; skew 500 ppm
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155134, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:14.511010 2212 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:14.511922 2212 webserver.cc:460] Webserver started at http://127.5.209.126:35675/ using document root <none> and password file <none>
I20260808 02:12:14.512532 2212 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:14.512643 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:14.512879 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:14.515069 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "07873bb234e244ec8f2563f4a94f6e9a"
format_stamp: "Formatted at 2026-08-08 02:12:14 on dist-test-slave-bw88"
server_key: "168ea91570ea20948ecfaa667a0c3e44"
server_key_iv: "7450757c9c45160cf1820579a9817090"
server_key_version: "encryptionkey@0"
I20260808 02:12:14.515659 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "07873bb234e244ec8f2563f4a94f6e9a"
format_stamp: "Formatted at 2026-08-08 02:12:14 on dist-test-slave-bw88"
server_key: "168ea91570ea20948ecfaa667a0c3e44"
server_key_iv: "7450757c9c45160cf1820579a9817090"
server_key_version: "encryptionkey@0"
I20260808 02:12:14.519510 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:14.521986 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:14.523170 2212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:12:14.523316 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "07873bb234e244ec8f2563f4a94f6e9a"
format_stamp: "Formatted at 2026-08-08 02:12:14 on dist-test-slave-bw88"
server_key: "168ea91570ea20948ecfaa667a0c3e44"
server_key_iv: "7450757c9c45160cf1820579a9817090"
server_key_version: "encryptionkey@0"
I20260808 02:12:14.523425 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:14.550931 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:14.554466 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:14.554683 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:14.561887 2212 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:38283
I20260808 02:12:14.561913 2306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:38283 every 8 connection(s)
I20260808 02:12:14.562863 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:14.565212 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2212
I20260808 02:12:14.565388 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:14.565742 5957 external_mini_cluster.cc:1428] Setting key 3ca4833f5ac00abea4e5804c5026146e
I20260808 02:12:14.565837 2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:14.573010 2308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a: Bootstrap starting.
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:14.576118 2308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:14.577083 2308 log.cc:824] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:14.579646 2308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a: No bootstrap required, opened a new log
I20260808 02:12:14.581318 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:14.567149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41839 (local address 127.5.209.126:38283)
0808 02:12:14.567656 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:14.567671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:14.567708 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:12:14.568621 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:14.568641 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:14.568996 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:14.569279 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:14.570290 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:14.571334 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:14.572506 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:14.572971 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:14.573107 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:14.576170 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:14.576178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:14.576196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:14.576229 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:14.578035 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:14.578570 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:14.578576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:14.578583 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:14.578723 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:14.579038 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:14.579041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:14.579043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:14.579550 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:14.579795 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:14.580139 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 02:12:14.580370 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":170,"threads_started":1}
I20260808 02:12:14.584182 2308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } }
I20260808 02:12:14.584446 2308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:14.584545 2308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07873bb234e244ec8f2563f4a94f6e9a, State: Initialized, Role: FOLLOWER
I20260808 02:12:14.584595 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:14.585035 2308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } }
I20260808 02:12:14.585191 2308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:14.585278 2308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:14.585397 2308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:14.586421 2308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } }
I20260808 02:12:14.586819 2308 leader_election.cc:302] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07873bb234e244ec8f2563f4a94f6e9a; no voters:
I20260808 02:12:14.587182 2308 leader_election.cc:288] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:14.587287 2322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:14.587563 2322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [term 1 LEADER]: Becoming Leader. State: Replica: 07873bb234e244ec8f2563f4a94f6e9a, State: Running, Role: LEADER
I20260808 02:12:14.587929 2322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } }
I20260808 02:12:14.588598 2308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:14.590014 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07873bb234e244ec8f2563f4a94f6e9a. Latest consensus state: current_term: 1 leader_uuid: "07873bb234e244ec8f2563f4a94f6e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } } }
I20260808 02:12:14.590058 2324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07873bb234e244ec8f2563f4a94f6e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07873bb234e244ec8f2563f4a94f6e9a" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 38283 } } }
I20260808 02:12:14.590142 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:14.590169 2324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:14.590559 2334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:14.594048 2334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:14.596340 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:14.586812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47039 (local address 127.5.209.126:38283)
0808 02:12:14.586933 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:14.586937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:14.587126 (+ 189us) server_negotiation.cc:407] Connection header received
0808 02:12:14.587291 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:14.587294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:14.587343 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:14.587424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:14.588378 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:14.589194 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:14.590533 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:14.590854 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:14.590960 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:14.592846 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:14.592850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:14.592853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:14.592877 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:14.594528 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:14.595010 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:14.595014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:14.595016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:14.595105 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:14.595446 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:14.595449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:14.595451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:14.595641 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:14.595798 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:14.596078 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 02:12:14.596179 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:12:14.602854 2334 catalog_manager.cc:1347] Generated new cluster ID: b92e1c6ade564d83bf32d9fb62102936
I20260808 02:12:14.602945 2334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:14.619997 2334 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:14.620983 2334 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:14.627991 2334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07873bb234e244ec8f2563f4a94f6e9a: Generated new TSK 0
I20260808 02:12:14.628815 2334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:14.634259 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2212
W20260808 02:12:14.646064 2193 connection.cc:535] client connection to 127.5.209.126:38283 recv error: Network error: failed to read from TLS socket (remote: 127.5.209.126:38283): Connection reset by peer (error 104)
2026-08-08T02:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2347](info): set up 2 sockets
Aug 08 02:12:14 dist-test-slave-bw88 krb5kdc[2347](info): commencing operation
krb5kdc: starting...
W20260808 02:12:15.823206 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:15 dist-test-slave-bw88 krb5kdc[2347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:15.960451 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35565
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:16.076946 2363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:16.077282 2363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:16.077395 2363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:16.081637 2363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:16.081740 2363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:16.081779 2363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:16.081833 2363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:16.081869 2363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:16.087471 2363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:35565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:16.088855 2363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:16.090076 2363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:16.099460 2370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:16.099452 2373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:16.099452 2369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:16.099824 2363 server_base.cc:1034] running on GCE node
I20260808 02:12:16.100473 2363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:16.101754 2363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:16.102959 2363 hybrid_clock.cc:648] HybridClock initialized: now 1786155136102944 us; error 29 us; skew 500 ppm
Aug 08 02:12:16 dist-test-slave-bw88 krb5kdc[2347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155136, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:16.105962 2363 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:16.106995 2363 webserver.cc:460] Webserver started at http://127.5.209.126:36381/ using document root <none> and password file <none>
I20260808 02:12:16.107613 2363 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:16.107709 2363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:16.107954 2363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:16.110210 2363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "bdf9e7a281604744a0e65432c265d3bf"
format_stamp: "Formatted at 2026-08-08 02:12:16 on dist-test-slave-bw88"
server_key: "7a447bf0494022be33b88893683ea0a8"
server_key_iv: "473be779347c67d3bf392b6a36a87bee"
server_key_version: "encryptionkey@0"
I20260808 02:12:16.110831 2363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "bdf9e7a281604744a0e65432c265d3bf"
format_stamp: "Formatted at 2026-08-08 02:12:16 on dist-test-slave-bw88"
server_key: "7a447bf0494022be33b88893683ea0a8"
server_key_iv: "473be779347c67d3bf392b6a36a87bee"
server_key_version: "encryptionkey@0"
I20260808 02:12:16.114821 2363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:12:16.117431 2381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:16.118790 2363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:12:16.118950 2363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "bdf9e7a281604744a0e65432c265d3bf"
format_stamp: "Formatted at 2026-08-08 02:12:16 on dist-test-slave-bw88"
server_key: "7a447bf0494022be33b88893683ea0a8"
server_key_iv: "473be779347c67d3bf392b6a36a87bee"
server_key_version: "encryptionkey@0"
I20260808 02:12:16.119071 2363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:16.138275 2363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:16.141574 2363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:16.141798 2363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:16.149605 2363 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:35565
I20260808 02:12:16.149680 2451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:35565 every 8 connection(s)
I20260808 02:12:16.150678 2363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:16.153918 2453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:16.158214 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2363
I20260808 02:12:16.158366 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:16.158669 5957 external_mini_cluster.cc:1428] Setting key 506e51da636a08941992a2b942148a82
I20260808 02:12:16.160197 2453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: Bootstrap starting.
I20260808 02:12:16.162616 2453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:16.163383 2453 log.cc:824] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:16.165462 2453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: No bootstrap required, opened a new log
I20260808 02:12:16.167929 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:16.160083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56349 (local address 127.5.209.126:35565)
0808 02:12:16.160551 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:16.160568 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:16.160606 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:16.161287 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:16.161305 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:16.161714 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:16.162054 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:16.162482 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:16.163608 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:16.164542 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:16.165026 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:16.165207 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:16.166204 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:16.166209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:16.166224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:16.166650 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:16.166707 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:16.166731 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:16.166877 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:16.167068 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:12:16.167215 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260808 02:12:16.169588 2453 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } }
I20260808 02:12:16.169833 2453 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:16.169902 2453 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf9e7a281604744a0e65432c265d3bf, State: Initialized, Role: FOLLOWER
I20260808 02:12:16.170418 2453 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } }
I20260808 02:12:16.170583 2453 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:16.170668 2453 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:16.170773 2453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:16.171197 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:16.171947 2453 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } }
I20260808 02:12:16.172336 2453 leader_election.cc:302] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf9e7a281604744a0e65432c265d3bf; no voters:
I20260808 02:12:16.172691 2453 leader_election.cc:288] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:16.172816 2463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:16.173121 2463 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [term 1 LEADER]: Becoming Leader. State: Replica: bdf9e7a281604744a0e65432c265d3bf, State: Running, Role: LEADER
I20260808 02:12:16.173573 2463 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } }
I20260808 02:12:16.174104 2453 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:16.175343 2470 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf9e7a281604744a0e65432c265d3bf. Latest consensus state: current_term: 1 leader_uuid: "bdf9e7a281604744a0e65432c265d3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } } }
I20260808 02:12:16.175498 2470 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:16.175720 2468 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf9e7a281604744a0e65432c265d3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf9e7a281604744a0e65432c265d3bf" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 35565 } } }
I20260808 02:12:16.175825 2468 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf [sys.catalog]: This master's current role is: LEADER
W20260808 02:12:16.179456 2491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:16.179554 2491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:16.179669 2490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:16.180478 2490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:16.181521 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:16.173643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45149 (local address 127.5.209.126:35565)
0808 02:12:16.173754 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:16.173759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:16.176824 (+ 3065us) server_negotiation.cc:407] Connection header received
0808 02:12:16.177012 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:16.177015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:16.177057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:16.177132 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:16.177469 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:16.178146 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:16.179338 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:16.179662 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:16.179765 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:16.180931 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:16.180935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:16.180939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:16.180955 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:16.180986 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:16.181001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:16.181096 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:16.181312 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 02:12:16.181381 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:12:16.187001 2490 catalog_manager.cc:1347] Generated new cluster ID: 3b00f9ea882b46368f6e834736c2da89
I20260808 02:12:16.187116 2490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:16.205878 2490 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:16.207345 2490 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:16.215173 2490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf9e7a281604744a0e65432c265d3bf: Generated new TSK 0
I20260808 02:12:16.216264 2490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:16.221671 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2363
W20260808 02:12:16.232641 2345 connection.cc:535] client connection to 127.5.209.126:35565 recv error: Network error: failed to read from TLS socket (remote: 127.5.209.126:35565): Connection reset by peer (error 104)
2026-08-08T02:12:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:16 dist-test-slave-bw88 krb5kdc[2500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:16 dist-test-slave-bw88 krb5kdc[2500](info): set up 2 sockets
Aug 08 02:12:16 dist-test-slave-bw88 krb5kdc[2500](info): commencing operation
krb5kdc: starting...
W20260808 02:12:17.425267 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:17 dist-test-slave-bw88 krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:17.558743 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:39763
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:39763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:17.714238 2518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:17.714550 2518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:17.714646 2518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:17.719298 2518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:17.719391 2518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:17.719429 2518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:17.719480 2518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:17.719517 2518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:17.725164 2518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:39763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:17.726363 2518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:17.727514 2518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:17.737681 2527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:17.737704 2528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:17.737887 2518 server_base.cc:1034] running on GCE node
W20260808 02:12:17.738040 2530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:17.738811 2518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:17.740492 2518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:17.741696 2518 hybrid_clock.cc:648] HybridClock initialized: now 1786155137741660 us; error 49 us; skew 500 ppm
Aug 08 02:12:17 dist-test-slave-bw88 krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155137, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:17.746330 2518 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:17.747303 2518 webserver.cc:460] Webserver started at http://127.5.209.126:37875/ using document root <none> and password file <none>
I20260808 02:12:17.747903 2518 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:17.747965 2518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:17.748217 2518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:17.750797 2518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7"
format_stamp: "Formatted at 2026-08-08 02:12:17 on dist-test-slave-bw88"
server_key: "b41eb665198204cf13e5050ec80e509c"
server_key_iv: "2d58373b5ee2bee7e010d4a506ef1264"
server_key_version: "encryptionkey@0"
I20260808 02:12:17.751384 2518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7"
format_stamp: "Formatted at 2026-08-08 02:12:17 on dist-test-slave-bw88"
server_key: "b41eb665198204cf13e5050ec80e509c"
server_key_iv: "2d58373b5ee2bee7e010d4a506ef1264"
server_key_version: "encryptionkey@0"
I20260808 02:12:17.755239 2518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:17.757799 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:17.759099 2518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 02:12:17.759240 2518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7"
format_stamp: "Formatted at 2026-08-08 02:12:17 on dist-test-slave-bw88"
server_key: "b41eb665198204cf13e5050ec80e509c"
server_key_iv: "2d58373b5ee2bee7e010d4a506ef1264"
server_key_version: "encryptionkey@0"
I20260808 02:12:17.759341 2518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:17.763988 2518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:17.766933 2518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:17.767139 2518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:17.774480 2518 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:39763
I20260808 02:12:17.774503 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:39763 every 8 connection(s)
I20260808 02:12:17.775455 2518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:17.777490 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2518
I20260808 02:12:17.777661 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:17.777952 5957 external_mini_cluster.cc:1428] Setting key 9e349c4f33a82ee539cf2f24e2247ab6
I20260808 02:12:17.778537 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:17.783277 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:17.779354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46089 (local address 127.5.209.126:39763)
0808 02:12:17.779814 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:17.779829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:17.779867 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:17.780535 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:17.780551 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:17.780841 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:17.781193 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:17.782039 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:17.782045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:17.782066 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:17.782078 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:17.782136 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:17.782157 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:17.782311 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:17.782385 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 02:12:17.782556 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260808 02:12:17.785209 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7: Bootstrap starting.
I20260808 02:12:17.786064 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:17.787945 2613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:17.788857 2613 log.cc:824] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:17.790171 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:17.788379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58165 (local address 127.5.209.126:39763)
0808 02:12:17.788507 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:17.788511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:17.788743 (+ 232us) server_negotiation.cc:407] Connection header received
0808 02:12:17.788869 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:17.788871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:17.788901 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:17.788988 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:17.789732 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:17.789738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:17.789744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:17.789760 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:17.789792 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:17.789809 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:17.789903 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:17.789992 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:12:17.790024 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:12:17.791386 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7: No bootstrap required, opened a new log
I20260808 02:12:17.795167 2613 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } }
I20260808 02:12:17.795383 2613 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:17.795418 2613 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc79d7d6a674a2bbcf69c323b7e64f7, State: Initialized, Role: FOLLOWER
I20260808 02:12:17.795774 2613 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } }
I20260808 02:12:17.795885 2613 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:17.795929 2613 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:17.796000 2613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:17.796831 2613 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } }
I20260808 02:12:17.797137 2613 leader_election.cc:302] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc79d7d6a674a2bbcf69c323b7e64f7; no voters:
I20260808 02:12:17.797444 2613 leader_election.cc:288] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:17.797586 2628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:17.797902 2628 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [term 1 LEADER]: Becoming Leader. State: Replica: dfc79d7d6a674a2bbcf69c323b7e64f7, State: Running, Role: LEADER
I20260808 02:12:17.798264 2628 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } }
I20260808 02:12:17.798694 2613 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:17.800086 2630 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } } }
I20260808 02:12:17.800243 2630 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:17.800629 2632 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc79d7d6a674a2bbcf69c323b7e64f7. Latest consensus state: current_term: 1 leader_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc79d7d6a674a2bbcf69c323b7e64f7" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 39763 } } }
I20260808 02:12:17.800746 2632 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:17.801074 2643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:17.803930 2643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:17.809768 2643 catalog_manager.cc:1347] Generated new cluster ID: 9729998d97844e3191ca5e25c3ad9a5e
I20260808 02:12:17.809839 2643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:17.821038 2643 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:17.821422 2643 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:17.825573 2643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc79d7d6a674a2bbcf69c323b7e64f7: Generated new TSK 0
I20260808 02:12:17.826280 2643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:17.834299 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2518
2026-08-08T02:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1614 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:17 dist-test-slave-bw88 krb5kdc[2656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:17 dist-test-slave-bw88 krb5kdc[2656](info): set up 2 sockets
Aug 08 02:12:17 dist-test-slave-bw88 krb5kdc[2656](info): commencing operation
krb5kdc: starting...
W20260808 02:12:19.023219 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:19.161412 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34685
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:19.275758 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:19.276010 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:19.276057 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:19.279911 2670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:19.279964 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:19.279983 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:19.280000 2670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:19.280015 2670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:19.285130 2670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:19.286228 2670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:19.287365 2670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:19.296363 2680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:19.296334 2678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:19.296389 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:19.296597 2670 server_base.cc:1034] running on GCE node
I20260808 02:12:19.297173 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:19.298321 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:19.299525 2670 hybrid_clock.cc:648] HybridClock initialized: now 1786155139299518 us; error 41 us; skew 500 ppm
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155139, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:19.302448 2670 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:19.303370 2670 webserver.cc:460] Webserver started at http://127.5.209.126:42805/ using document root <none> and password file <none>
I20260808 02:12:19.303941 2670 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:19.304030 2670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:19.304265 2670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:19.306344 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "793a6a38e6f841c8999e4d33196df5b0"
format_stamp: "Formatted at 2026-08-08 02:12:19 on dist-test-slave-bw88"
server_key: "dea7a0b70d8559f703a1a61b27f14d9d"
server_key_iv: "1bcca0c6259ba61950f87cf72c56e90a"
server_key_version: "encryptionkey@0"
I20260808 02:12:19.306915 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "793a6a38e6f841c8999e4d33196df5b0"
format_stamp: "Formatted at 2026-08-08 02:12:19 on dist-test-slave-bw88"
server_key: "dea7a0b70d8559f703a1a61b27f14d9d"
server_key_iv: "1bcca0c6259ba61950f87cf72c56e90a"
server_key_version: "encryptionkey@0"
I20260808 02:12:19.310745 2670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:12:19.313102 2691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:19.314370 2670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 02:12:19.314514 2670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "793a6a38e6f841c8999e4d33196df5b0"
format_stamp: "Formatted at 2026-08-08 02:12:19 on dist-test-slave-bw88"
server_key: "dea7a0b70d8559f703a1a61b27f14d9d"
server_key_iv: "1bcca0c6259ba61950f87cf72c56e90a"
server_key_version: "encryptionkey@0"
I20260808 02:12:19.314648 2670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:19.328728 2670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:19.331945 2670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:19.332149 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:19.339699 2670 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:34685
I20260808 02:12:19.339728 2762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:34685 every 8 connection(s)
I20260808 02:12:19.340662 2670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:19.343557 2764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:19.348606 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2670
I20260808 02:12:19.348757 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:19.349076 5957 external_mini_cluster.cc:1428] Setting key f48d8a9d27af73dd298b8c310ddb67b7
I20260808 02:12:19.349463 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0: Bootstrap starting.
I20260808 02:12:19.352010 2764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:19.352766 2764 log.cc:824] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:19.355151 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0: No bootstrap required, opened a new log
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:19.359478 2764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } }
I20260808 02:12:19.359733 2764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:19.359814 2764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a6a38e6f841c8999e4d33196df5b0, State: Initialized, Role: FOLLOWER
I20260808 02:12:19.360409 2764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } }
I20260808 02:12:19.360600 2764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:19.360675 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:19.360786 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:19.361972 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } }
I20260808 02:12:19.362362 2764 leader_election.cc:302] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793a6a38e6f841c8999e4d33196df5b0; no voters:
I20260808 02:12:19.362730 2764 leader_election.cc:288] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:19.362808 2769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:19.363023 2769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [term 1 LEADER]: Becoming Leader. State: Replica: 793a6a38e6f841c8999e4d33196df5b0, State: Running, Role: LEADER
I20260808 02:12:19.363366 2769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } }
I20260808 02:12:19.364086 2764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:19.364588 2769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793a6a38e6f841c8999e4d33196df5b0. Latest consensus state: current_term: 1 leader_uuid: "793a6a38e6f841c8999e4d33196df5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } } }
I20260808 02:12:19.364706 2769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:19.365027 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:19.350517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48247 (local address 127.5.209.126:34685)
0808 02:12:19.350976 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:19.350991 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:19.351026 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:12:19.351657 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:19.351674 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:19.352006 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:19.352364 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:19.353240 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:19.354698 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:19.355636 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:19.356310 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:19.356476 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:19.359272 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:19.359277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:19.359290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:19.359316 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:19.361446 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:19.361881 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:19.361889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:19.361898 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:19.362058 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:19.362395 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:19.362398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:19.362399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:19.362991 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:19.363213 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:19.363714 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 02:12:19.364024 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":152,"threads_started":1}
I20260808 02:12:19.365430 2770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793a6a38e6f841c8999e4d33196df5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793a6a38e6f841c8999e4d33196df5b0" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 34685 } } }
I20260808 02:12:19.365542 2770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:19.365861 2776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:19.368396 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:19.369197 2776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:19.375663 2776 catalog_manager.cc:1347] Generated new cluster ID: c9d487d05a5d4a9c8a1109b5b278096b
I20260808 02:12:19.375739 2776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:19.380549 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:19.371752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48373 (local address 127.5.209.126:34685)
0808 02:12:19.371854 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:19.371858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:19.372190 (+ 332us) server_negotiation.cc:407] Connection header received
0808 02:12:19.372349 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:19.372352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:19.372404 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:19.372495 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:19.373414 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:19.374087 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:19.374905 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:19.375168 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:19.375277 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:19.376876 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:19.376880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:19.376883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:19.376912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:19.378631 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:19.379094 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:19.379098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:19.379100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:19.379191 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:19.379470 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:19.379473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:19.379475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:19.379710 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:19.379864 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:19.380095 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 02:12:19.380221 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 02:12:19.383769 2776 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:19.384618 2776 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:19.390887 2776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793a6a38e6f841c8999e4d33196df5b0: Generated new TSK 0
I20260808 02:12:19.391578 2776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:19.397420 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2670
2026-08-08T02:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2805](info): set up 2 sockets
Aug 08 02:12:19 dist-test-slave-bw88 krb5kdc[2805](info): commencing operation
krb5kdc: starting...
W20260808 02:12:20.602353 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:20 dist-test-slave-bw88 krb5kdc[2805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:20.736191 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43315
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:20.856323 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:20.856613 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:20.856655 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:20.860713 2821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:20.860775 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:20.860795 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:20.860811 2821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:20.860848 2821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:20.866226 2821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:20.867277 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:20.868328 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:20.877368 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:20.877445 2821 server_base.cc:1034] running on GCE node
W20260808 02:12:20.877329 2829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:20.877399 2828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:20.878149 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:20.879258 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:20.880450 2821 hybrid_clock.cc:648] HybridClock initialized: now 1786155140880434 us; error 30 us; skew 500 ppm
Aug 08 02:12:20 dist-test-slave-bw88 krb5kdc[2805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155140, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:20.883229 2821 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:20.884084 2821 webserver.cc:460] Webserver started at http://127.5.209.126:35243/ using document root <none> and password file <none>
I20260808 02:12:20.884670 2821 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:20.884735 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:20.884909 2821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:20.887063 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "eeeaed01cff149b5b3a0874c99688744"
format_stamp: "Formatted at 2026-08-08 02:12:20 on dist-test-slave-bw88"
server_key: "96e6f6ff7a972726e432d29f6ea3d6fe"
server_key_iv: "c1fc6596477fef0ef8700e3caea0b354"
server_key_version: "encryptionkey@0"
I20260808 02:12:20.887612 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "eeeaed01cff149b5b3a0874c99688744"
format_stamp: "Formatted at 2026-08-08 02:12:20 on dist-test-slave-bw88"
server_key: "96e6f6ff7a972726e432d29f6ea3d6fe"
server_key_iv: "c1fc6596477fef0ef8700e3caea0b354"
server_key_version: "encryptionkey@0"
I20260808 02:12:20.891752 2821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:20.894413 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:20.895731 2821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:20.895839 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "eeeaed01cff149b5b3a0874c99688744"
format_stamp: "Formatted at 2026-08-08 02:12:20 on dist-test-slave-bw88"
server_key: "96e6f6ff7a972726e432d29f6ea3d6fe"
server_key_iv: "c1fc6596477fef0ef8700e3caea0b354"
server_key_version: "encryptionkey@0"
I20260808 02:12:20.895929 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:20.908608 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:20.912099 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:20.912268 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:20.919970 2821 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:43315
I20260808 02:12:20.919987 2910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:43315 every 8 connection(s)
I20260808 02:12:20.921008 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:20.923455 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2821
I20260808 02:12:20.923568 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:20.923877 5957 external_mini_cluster.cc:1428] Setting key bcccdcd550bd0d0cce18f8b54489fcd4
I20260808 02:12:20.924098 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:20.931824 2911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: Bootstrap starting.
I20260808 02:12:20.935072 2911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:20.935184 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:20.925454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57235 (local address 127.5.209.126:43315)
0808 02:12:20.925930 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:20.925950 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:20.926019 (+ 69us) server_negotiation.cc:407] Connection header received
0808 02:12:20.927116 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:20.927143 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:20.927623 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:20.928005 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:20.928443 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:20.929972 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:20.931030 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:20.931738 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:20.931910 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:20.932972 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:20.932992 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:20.933008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:20.933588 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:20.933663 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:20.933697 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:20.933806 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:20.933966 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:12:20.934192 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260808 02:12:20.935925 2911 log.cc:824] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:20.938508 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:20.938520 2911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: No bootstrap required, opened a new log
I20260808 02:12:20.943132 2911 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } }
I20260808 02:12:20.943434 2911 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:20.943576 2911 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeeaed01cff149b5b3a0874c99688744, State: Initialized, Role: FOLLOWER
I20260808 02:12:20.944121 2911 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } }
I20260808 02:12:20.944293 2911 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:20.944387 2911 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:20.944518 2911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:20.944917 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:20.940716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54473 (local address 127.5.209.126:43315)
0808 02:12:20.940865 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:20.940870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:20.940973 (+ 103us) server_negotiation.cc:407] Connection header received
0808 02:12:20.941139 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:20.941151 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:20.941203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:20.941294 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:20.941636 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:20.942214 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:20.943119 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:20.943372 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:20.943494 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:20.944447 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:20.944451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:20.944454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:20.944471 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:20.944502 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:20.944516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:20.944610 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:20.944713 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:12:20.944771 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:12:20.945760 2911 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } }
I20260808 02:12:20.946151 2911 leader_election.cc:302] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeeaed01cff149b5b3a0874c99688744; no voters:
I20260808 02:12:20.946519 2911 leader_election.cc:288] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:20.946748 2923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:20.947032 2923 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [term 1 LEADER]: Becoming Leader. State: Replica: eeeaed01cff149b5b3a0874c99688744, State: Running, Role: LEADER
I20260808 02:12:20.947389 2923 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } }
I20260808 02:12:20.948004 2911 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:20.949357 2926 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeeaed01cff149b5b3a0874c99688744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } } }
I20260808 02:12:20.949519 2926 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:20.949801 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeeaed01cff149b5b3a0874c99688744. Latest consensus state: current_term: 1 leader_uuid: "eeeaed01cff149b5b3a0874c99688744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeeaed01cff149b5b3a0874c99688744" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 43315 } } }
I20260808 02:12:20.949945 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744 [sys.catalog]: This master's current role is: LEADER
W20260808 02:12:20.953521 2944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:20.953610 2944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:20.953688 2937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:20.954409 2937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:20.960084 2937 catalog_manager.cc:1347] Generated new cluster ID: 3cd85709cbf94fea91f5974ed2ba7c66
I20260808 02:12:20.960162 2937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:20.979766 2937 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:20.981292 2937 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:20.986502 2937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeeaed01cff149b5b3a0874c99688744: Generated new TSK 0
I20260808 02:12:20.987259 2937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:20.991070 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2821
2026-08-08T02:12:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1590 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:21 dist-test-slave-bw88 krb5kdc[2955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:21 dist-test-slave-bw88 krb5kdc[2955](info): set up 2 sockets
Aug 08 02:12:21 dist-test-slave-bw88 krb5kdc[2955](info): commencing operation
krb5kdc: starting...
W20260808 02:12:22.186936 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:22 dist-test-slave-bw88 krb5kdc[2955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:22.326370 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42845
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:22.446947 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:22.447319 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:22.447405 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:22.451571 2974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:22.451685 2974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:22.451725 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:22.451761 2974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:22.451788 2974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:22.457208 2974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:22.458504 2974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:22.459702 2974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:22.468811 2984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:22.468888 2974 server_base.cc:1034] running on GCE node
W20260808 02:12:22.468842 2983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:22.468811 2986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:22.469748 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:22.470902 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:22.472111 2974 hybrid_clock.cc:648] HybridClock initialized: now 1786155142472076 us; error 48 us; skew 500 ppm
Aug 08 02:12:22 dist-test-slave-bw88 krb5kdc[2955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155142, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:22.475158 2974 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:22.476014 2974 webserver.cc:460] Webserver started at http://127.5.209.126:44717/ using document root <none> and password file <none>
I20260808 02:12:22.476563 2974 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:22.476624 2974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:22.476794 2974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:22.478916 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "e13897a8f93e49af8eb0f80dea74fe62"
format_stamp: "Formatted at 2026-08-08 02:12:22 on dist-test-slave-bw88"
server_key: "60b53d7dbc069fb236619e5ed3098485"
server_key_iv: "4f8d03d84986ec637249bea447ae6223"
server_key_version: "encryptionkey@0"
I20260808 02:12:22.479445 2974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "e13897a8f93e49af8eb0f80dea74fe62"
format_stamp: "Formatted at 2026-08-08 02:12:22 on dist-test-slave-bw88"
server_key: "60b53d7dbc069fb236619e5ed3098485"
server_key_iv: "4f8d03d84986ec637249bea447ae6223"
server_key_version: "encryptionkey@0"
I20260808 02:12:22.483297 2974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:22.485875 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:22.487236 2974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:22.487376 2974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "e13897a8f93e49af8eb0f80dea74fe62"
format_stamp: "Formatted at 2026-08-08 02:12:22 on dist-test-slave-bw88"
server_key: "60b53d7dbc069fb236619e5ed3098485"
server_key_iv: "4f8d03d84986ec637249bea447ae6223"
server_key_version: "encryptionkey@0"
I20260808 02:12:22.487488 2974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:22.492246 2974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:22.495810 2974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:22.496017 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:22.503582 2974 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:42845
I20260808 02:12:22.503607 3065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:42845 every 8 connection(s)
I20260808 02:12:22.504616 2974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:22.507583 3066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:22.513396 3066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: Bootstrap starting.
I20260808 02:12:22.514328 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 2974
I20260808 02:12:22.514470 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:22.514824 5957 external_mini_cluster.cc:1428] Setting key 4a9f1757962cb5981c4bb474f923aeaf
I20260808 02:12:22.516518 3066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:22.517395 3066 log.cc:824] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:22.519855 3066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: No bootstrap required, opened a new log
I20260808 02:12:22.520354 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:22.515993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51281 (local address 127.5.209.126:42845)
0808 02:12:22.516523 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:22.516540 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:22.516606 (+ 66us) server_negotiation.cc:407] Connection header received
0808 02:12:22.517297 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:22.517315 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:22.517724 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:22.518137 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:22.519054 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:22.519059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:22.519072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:22.519083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:22.519139 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:22.519159 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:22.519285 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:22.519388 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 02:12:22.519561 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260808 02:12:22.523545 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:22.524303 3066 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } }
I20260808 02:12:22.524518 3066 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:12:22.524585 3066 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13897a8f93e49af8eb0f80dea74fe62, State: Initialized, Role: FOLLOWER
I20260808 02:12:22.525075 3066 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } }
I20260808 02:12:22.525207 3066 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:12:22.525301 3066 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:12:22.525449 3066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:22.526767 3066 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } }
I20260808 02:12:22.527194 3066 leader_election.cc:302] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13897a8f93e49af8eb0f80dea74fe62; no voters:
I20260808 02:12:22.527572 3066 leader_election.cc:288] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:12:22.527591 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:22.525855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51195 (local address 127.5.209.126:42845)
0808 02:12:22.525984 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:22.525988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:22.526133 (+ 145us) server_negotiation.cc:407] Connection header received
0808 02:12:22.526232 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:22.526235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:22.526280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:22.526356 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:22.527140 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:22.527144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:22.527147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:12:22.527159 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:22.527189 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:12:22.527204 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:22.527300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:22.527429 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:12:22.527468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:12:22.527693 3079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:22.527923 3079 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [term 1 LEADER]: Becoming Leader. State: Replica: e13897a8f93e49af8eb0f80dea74fe62, State: Running, Role: LEADER
I20260808 02:12:22.528307 3079 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } }
I20260808 02:12:22.528988 3066 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:22.529510 3079 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13897a8f93e49af8eb0f80dea74fe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } } }
I20260808 02:12:22.529681 3079 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:22.529955 3081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13897a8f93e49af8eb0f80dea74fe62. Latest consensus state: current_term: 1 leader_uuid: "e13897a8f93e49af8eb0f80dea74fe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13897a8f93e49af8eb0f80dea74fe62" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 42845 } } }
I20260808 02:12:22.530066 3081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62 [sys.catalog]: This master's current role is: LEADER
W20260808 02:12:22.534195 3100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:22.534276 3100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:22.534454 3089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:22.535207 3089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:22.540925 3089 catalog_manager.cc:1347] Generated new cluster ID: 44423a970f0e4a5b975817ef1326e29c
I20260808 02:12:22.541026 3089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:22.549255 3089 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:22.549609 3089 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:22.561223 3089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13897a8f93e49af8eb0f80dea74fe62: Generated new TSK 0
I20260808 02:12:22.562021 3089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:12:22.569417 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 2974
2026-08-08T02:12:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1582 ms)
[----------] 18 tests from AuthTokenIssuingTest (28785 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:22 dist-test-slave-bw88 krb5kdc[3108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:22 dist-test-slave-bw88 krb5kdc[3108](info): set up 2 sockets
Aug 08 02:12:22 dist-test-slave-bw88 krb5kdc[3108](info): commencing operation
krb5kdc: starting...
W20260808 02:12:23.815670 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:23 dist-test-slave-bw88 krb5kdc[3108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:23.961452 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:44611
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:24.082388 3127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:24.082768 3127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:24.082849 3127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:24.082921 3127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:24.087220 3127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:24.087311 3127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:24.087350 3127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:24.087409 3127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:24.087458 3127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:24.092926 3127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:44611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:24.094182 3127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:24.095288 3127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:24.104696 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:24.104740 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:24.104879 3127 server_base.cc:1034] running on GCE node
W20260808 02:12:24.104704 3135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:24.105553 3127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:24.106732 3127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:24.107954 3127 hybrid_clock.cc:648] HybridClock initialized: now 1786155144107941 us; error 29 us; skew 500 ppm
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:24.111050 3127 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:24.111943 3127 webserver.cc:460] Webserver started at http://127.5.209.126:35489/ using document root <none> and password file <none>
I20260808 02:12:24.112505 3127 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:24.112576 3127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:24.112747 3127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:24.114864 3127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "9eba79264d6741418a52921bb603e622"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "e6a9a2615be415c1160c9b2ca5cfe4be"
server_key_iv: "1675703f1f5350ca8f0c30fe64a710ad"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.115386 3127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "9eba79264d6741418a52921bb603e622"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "e6a9a2615be415c1160c9b2ca5cfe4be"
server_key_iv: "1675703f1f5350ca8f0c30fe64a710ad"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.119315 3127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:24.121966 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:24.123543 3127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:24.123725 3127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "9eba79264d6741418a52921bb603e622"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "e6a9a2615be415c1160c9b2ca5cfe4be"
server_key_iv: "1675703f1f5350ca8f0c30fe64a710ad"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.123845 3127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:24.135818 3127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:24.139276 3127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:24.139557 3127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:24.147217 3127 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:44611
I20260808 02:12:24.147523 3211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:44611 every 8 connection(s)
I20260808 02:12:24.148341 3127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:24.149734 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3127
I20260808 02:12:24.149886 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:24.150199 5957 external_mini_cluster.cc:1428] Setting key cc83884b71ce3feb3c26b1068fe5ce94
I20260808 02:12:24.151889 3214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:24.157812 3214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.209.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.125@KRBTEST.COM" created.
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:24.174949 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.160486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:44611 (local address 127.0.0.1:43840)
0808 02:12:24.161086 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.161130 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.162224 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.162942 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.162948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.163400 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.163674 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.163688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.164944 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.164947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.165787 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.165797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.166628 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.166632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.166751 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.167435 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.167455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.169133 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.171811 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.171820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.171829 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.172085 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.172499 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.172501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.172502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.172614 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.173519 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.173525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.173657 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.173806 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.174090 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":4,"thread_start_us":177,"threads_started":1}
I20260808 02:12:24.174968 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.161585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.5.209.126:44611)
0808 02:12:24.161888 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.161899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.161947 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:12:24.162553 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.162558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.162743 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.162945 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.163833 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.164810 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.165915 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.166441 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.166630 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.169320 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.169332 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.169381 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.169419 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.171674 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.172216 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.172225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.172227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.172388 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.172724 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.172729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.172732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.173384 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.173544 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.173973 (+ 429us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.174179 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260808 02:12:24.177841 3214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:24.178690 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.178391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39300)
0808 02:12:24.178550 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.178627 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 02:12:24.179076 3149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
I20260808 02:12:24.180055 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.179841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39310)
0808 02:12:24.179954 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.180001 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:12:24.180315 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:37361: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
I20260808 02:12:24.184923 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:37361
--webserver_interface=127.5.209.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:12:24.229610 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } attempt: 1
I20260808 02:12:24.230665 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.230264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39318)
0808 02:12:24.230469 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.230574 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:12:24.231552 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.231304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39334)
0808 02:12:24.231415 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.231461 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:12:24.231747 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:37361: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
I20260808 02:12:24.299925 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } attempt: 2
I20260808 02:12:24.301024 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.300595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39338)
0808 02:12:24.300797 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.300904 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:12:24.302075 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.301810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39348)
0808 02:12:24.301960 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.302013 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 02:12:24.302322 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:37361: Network error: Client connection negotiation failed: client connection to 127.5.209.125:37361: connect: Connection refused (error 111)
W20260808 02:12:24.307505 3222 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:24.307796 3222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:24.307878 3222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:24.307952 3222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:24.312316 3222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:24.312371 3222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:24.312390 3222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:24.312440 3222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:24.312479 3222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:24.318050 3222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.209.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:24.319285 3222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:24.320446 3222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:24.330281 3230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:24.330317 3227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:24.330317 3228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:24.330514 3222 server_base.cc:1034] running on GCE node
I20260808 02:12:24.331117 3222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:24.332332 3222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:24.333626 3222 hybrid_clock.cc:648] HybridClock initialized: now 1786155144333594 us; error 46 us; skew 500 ppm
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:24.336779 3222 init.cc:377] Logged in from keytab as kudu/127.5.209.125@KRBTEST.COM (short username kudu)
I20260808 02:12:24.337889 3222 webserver.cc:460] Webserver started at http://127.5.209.125:33687/ using document root <none> and password file <none>
I20260808 02:12:24.338515 3222 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:24.338626 3222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:24.338891 3222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:24.341097 3222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data/instance:
uuid: "f8ad32c83394452b8734e8fde06f338e"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "7153f2023e8a0c72aee58313a9ec5491"
server_key_iv: "4e187839a2b2cfbb6eea9293b6659805"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.341817 3222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal/instance:
uuid: "f8ad32c83394452b8734e8fde06f338e"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "7153f2023e8a0c72aee58313a9ec5491"
server_key_iv: "4e187839a2b2cfbb6eea9293b6659805"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.345875 3222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:12:24.348503 3237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:24.349921 3222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:24.350090 3222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "f8ad32c83394452b8734e8fde06f338e"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "7153f2023e8a0c72aee58313a9ec5491"
server_key_iv: "4e187839a2b2cfbb6eea9293b6659805"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.350220 3222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:24.380456 3222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:24.383966 3222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:24.384224 3222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:24.392596 3222 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.125:37361
I20260808 02:12:24.392903 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.125:37361 every 8 connection(s)
I20260808 02:12:24.393532 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } attempt: 3
I20260808 02:12:24.393858 3222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
I20260808 02:12:24.397522 3306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:24.402702 3306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:24.403770 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3222
I20260808 02:12:24.403918 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-1/wal/instance
I20260808 02:12:24.404276 5957 external_mini_cluster.cc:1428] Setting key 5b79d82814a0265884cfa93983c67ebb
I20260808 02:12:24.408444 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.394172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39362)
0808 02:12:24.394347 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.394366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.394481 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.397149 (+ 2668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.397152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.397324 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.397551 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.397560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.399368 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.399372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.400070 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.400082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.401099 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.401131 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.401284 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.401956 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.401969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.403535 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.406237 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.406241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.406243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.406533 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.406943 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.406948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.406951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.407065 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.407891 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.407894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.407997 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.408121 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.408266 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 02:12:24.409569 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.394934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.5.209.125:37361)
0808 02:12:24.395370 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.395391 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.395436 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:12:24.396341 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.396364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.396806 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.397182 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.397740 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.399217 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.400220 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.400938 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.401109 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.403766 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.403772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.403788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.403820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.406119 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.406667 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.406673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.406679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.406811 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.407198 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.407204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.407207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.407785 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.407932 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.408313 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.408636 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
I20260808 02:12:24.411904 3214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:24.412734 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.412462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36528)
0808 02:12:24.412614 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.412666 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:24.413635 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.413301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36538)
0808 02:12:24.413498 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.413561 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260808 02:12:24.413887 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.209.124@KRBTEST.COM; defaulting to no policy
I20260808 02:12:24.417332 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.5.209.126:44611)
0808 02:12:24.406133 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.406140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.406156 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:12:24.406197 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.406203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.406256 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.406345 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.407156 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.407680 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.408862 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.410103 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.410228 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.413747 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.413751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.413753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.413778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.415360 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.415979 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.415984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.415986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.416088 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.416396 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.416400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.416402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.416606 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.416721 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.417133 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.417235 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:12:24.417454 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.404937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:44611 (local address 127.0.0.1:43842)
0808 02:12:24.405500 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.405550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.405667 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.406399 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.406405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.406864 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.407034 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.407040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.407820 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.407824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.408731 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.408741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.410241 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.410245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.410374 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.411718 (+ 1344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.411732 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.413605 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.415535 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.415547 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.415553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.415854 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.416193 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.416197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.416199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.416294 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.416741 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.416748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.416873 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.417051 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.417237 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":182,"threads_started":1}
I20260808 02:12:24.418617 3306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.124@KRBTEST.COM" created.
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:24.428216 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.419249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39376)
0808 02:12:24.419376 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.419390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.419471 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.419810 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.419813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.419948 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.420047 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.420052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.421097 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.421100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.422038 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.422052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.422515 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.422518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.422652 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.423146 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.423166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.424560 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.426598 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.426601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.426602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.426796 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.427122 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.427125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.427126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.427240 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.427734 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.427736 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.427820 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.427904 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.428016 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 02:12:24.428442 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.419415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.5.209.125:37361)
0808 02:12:24.419564 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.419569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.419586 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:12:24.419626 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.419629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.419680 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.419765 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.420169 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.420975 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.422174 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.422393 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.422476 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.424710 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.424717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.424723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.424756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.426470 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.426908 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.426914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.426919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.427024 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.427344 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.427350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.427354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.427631 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.427738 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.428005 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.428143 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:12:24.429383 3306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:24.430202 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.429845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36552)
0808 02:12:24.430010 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.430124 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:12:24.430825 3243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
I20260808 02:12:24.431856 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.431545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36560)
0808 02:12:24.431717 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.431778 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 02:12:24.432008 3306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
I20260808 02:12:24.441548 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:45561
--webserver_interface=127.5.209.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:12:24.445075 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 1
I20260808 02:12:24.446148 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.445709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36564)
0808 02:12:24.445926 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.446027 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:12:24.447079 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.446740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36574)
0808 02:12:24.446906 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.446983 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:12:24.447230 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
I20260808 02:12:24.481371 3306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 1
I20260808 02:12:24.482367 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.482027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36588)
0808 02:12:24.482200 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.482272 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 02:12:24.483283 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.483057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36596)
0808 02:12:24.483173 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.483223 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:12:24.483472 3306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
I20260808 02:12:24.522469 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 2
I20260808 02:12:24.523542 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.523149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36612)
0808 02:12:24.523330 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.523419 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:12:24.524425 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.524208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36622)
0808 02:12:24.524325 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.524374 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 02:12:24.524704 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
I20260808 02:12:24.551698 3306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 2
I20260808 02:12:24.552714 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.552372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36624)
0808 02:12:24.552560 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.552631 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:12:24.553643 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.553408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36626)
0808 02:12:24.553528 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.553583 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 02:12:24.553787 3306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
W20260808 02:12:24.569319 3315 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:24.569706 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:24.569773 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:24.569847 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:24.573879 3315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:24.573967 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:24.574004 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:24.574038 3315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:24.574065 3315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:24.579610 3315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:44611,127.5.209.125:37361,127.5.209.124:45561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.209.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:24.580834 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:24.582042 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:24.591667 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:24.591667 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:24.591711 3315 server_base.cc:1034] running on GCE node
W20260808 02:12:24.591668 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:24.592617 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:24.593843 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:24.595057 3315 hybrid_clock.cc:648] HybridClock initialized: now 1786155144595039 us; error 28 us; skew 500 ppm
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:24.598109 3315 init.cc:377] Logged in from keytab as kudu/127.5.209.124@KRBTEST.COM (short username kudu)
I20260808 02:12:24.599064 3315 webserver.cc:460] Webserver started at http://127.5.209.124:40309/ using document root <none> and password file <none>
I20260808 02:12:24.599620 3315 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:24.599677 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:24.599926 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:24.602200 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data/instance:
uuid: "5e257c215cd34dc3956449d80cda054b"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "524daa718dea3a6750f0b1b205042b10"
server_key_iv: "57918570c7fa7ccdbddc3a1ed31c162f"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.602730 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal/instance:
uuid: "5e257c215cd34dc3956449d80cda054b"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "524daa718dea3a6750f0b1b205042b10"
server_key_iv: "57918570c7fa7ccdbddc3a1ed31c162f"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.606546 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:12:24.609118 3332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:24.610620 3315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:24.610764 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "5e257c215cd34dc3956449d80cda054b"
format_stamp: "Formatted at 2026-08-08 02:12:24 on dist-test-slave-bw88"
server_key: "524daa718dea3a6750f0b1b205042b10"
server_key_iv: "57918570c7fa7ccdbddc3a1ed31c162f"
server_key_version: "encryptionkey@0"
I20260808 02:12:24.610880 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:24.623889 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 3
I20260808 02:12:24.624768 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.624452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36642)
0808 02:12:24.624623 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.624684 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:12:24.625437 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:24.625572 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.625294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36648)
0808 02:12:24.625431 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.625487 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 02:12:24.625728 3214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45561: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45561: connect: Connection refused (error 111)
I20260808 02:12:24.628912 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:24.629123 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:24.636634 3315 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.124:45561
I20260808 02:12:24.636940 3399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.124:45561 every 8 connection(s)
I20260808 02:12:24.637735 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
I20260808 02:12:24.640635 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3315
I20260808 02:12:24.640681 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:24.640779 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786154850122309-5957-0/minicluster-data/master-2/wal/instance
I20260808 02:12:24.641059 5957 external_mini_cluster.cc:1428] Setting key 7867805ba7c0104d7ada9b982f2e013a
I20260808 02:12:24.645001 3306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 3
I20260808 02:12:24.645838 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:24.657081 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.645561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36656)
0808 02:12:24.645783 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.645800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.645893 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.646197 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.646203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.646413 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.646505 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.646511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.647381 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.647387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.648207 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.648218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.649264 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.649268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.649383 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.650540 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.650552 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.651863 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.654542 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.654546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.654548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.654830 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.655271 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.655274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.655277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.655372 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.656183 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.656189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.656300 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.656417 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.656576 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260808 02:12:24.657991 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.645590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.5.209.124:45561)
0808 02:12:24.645962 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.645974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.645986 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:12:24.646017 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.646019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.646066 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.646171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.646630 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.647244 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.648342 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.649134 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.649303 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.652050 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.652056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.652065 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.652093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.654392 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.654971 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.654980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.654988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.655148 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.655487 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.655490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.655492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.656060 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.656169 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.656607 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.656907 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":296,"spinlock_wait_cycles":6784,"thread_start_us":150,"threads_started":1}
I20260808 02:12:24.657991 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.642624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.5.209.124:45561)
0808 02:12:24.643049 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.643065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.643103 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:24.644106 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.644127 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.644453 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.644738 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.645753 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.647248 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.648375 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.649136 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.649298 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.652760 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.652766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.652770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.652801 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.654932 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.655487 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.655490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.655492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.655640 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.655933 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.655936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.655938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.656146 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.656327 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.656872 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.656958 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":7424,"thread_start_us":140,"threads_started":1}
I20260808 02:12:24.659824 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.647958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.5.209.126:44611)
0808 02:12:24.648079 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.648084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.648511 (+ 427us) server_negotiation.cc:407] Connection header received
0808 02:12:24.648729 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.648732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.648793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.648880 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.649775 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.650399 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.651541 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.651919 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.652022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.655443 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.655447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.655449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.655479 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.657557 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.658102 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.658106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.658107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.658190 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.658909 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.658913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.658915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.659140 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.659225 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.659622 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.659717 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:12:24.659863 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.647860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:44611 (local address 127.0.0.1:43844)
0808 02:12:24.648391 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.648442 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.648599 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.648961 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.648967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.649486 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.649633 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.649641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.650532 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.650537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.651410 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.651421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.652168 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.652173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.652298 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.652946 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.652961 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.655293 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.657671 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.657683 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.657691 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.657999 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.658688 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.658691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.658692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.658786 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.659254 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.659260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.659374 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.659521 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.659689 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":99,"threads_started":1}
I20260808 02:12:24.661106 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:24.663183 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: Bootstrap starting.
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:24.666396 3306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: Neither blocks nor log segments found. Creating new log.
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:24.667295 3306 log.cc:824] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:24.670307 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: No bootstrap required, opened a new log
I20260808 02:12:24.670917 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.661607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:37361 (local address 127.0.0.1:39390)
0808 02:12:24.661761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.661776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.661845 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.662085 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.662088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.662263 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.662401 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.662412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.663377 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.663381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.664128 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.664140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.664684 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.664688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.664790 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.665295 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.665306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.667144 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.669168 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.669171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.669174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.669467 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.669830 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.669833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.669835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.669934 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.670433 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.670436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.670528 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.670609 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.670760 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:12:24.671162 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.661019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.5.209.126:44611)
0808 02:12:24.661453 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.661459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.661473 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:12:24.661510 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.661516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.661561 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.661838 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.662696 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.663162 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.663995 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.664273 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.664371 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.667348 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.667351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.667353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.667374 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.669154 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.669653 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.669659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.669663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.669741 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.670046 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.670050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.670052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.670250 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.670494 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.670731 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.670849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260808 02:12:24.671427 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.661706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.5.209.125:37361)
0808 02:12:24.661865 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.661870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.661884 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:12:24.661917 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.661920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.661960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.662052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.662543 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.663242 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.664260 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.664562 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.664679 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.667295 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.667299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.667303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.667333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.669034 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.669605 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.669610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.669615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.669714 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.670066 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.670071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.670073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.670308 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.671106 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.671184 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.671296 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:12:24.672201 3401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:24.674911 3306 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.675102 3306 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ad32c83394452b8734e8fde06f338e, State: Initialized, Role: FOLLOWER
I20260808 02:12:24.675472 3306 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.676079 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.676262 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.677322 3306 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:24.682114 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.672753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36672)
0808 02:12:24.672869 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.672884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.672975 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.673222 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.673225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.673454 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.673552 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.673562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.674298 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.674304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.675076 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.675087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.675676 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.675679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.675762 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.676351 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.676367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.677954 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.680028 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.680030 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.680034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.680247 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.680782 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.680788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.680791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.680888 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.681597 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.681600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.681700 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.681819 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.681949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:12:24.682548 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.672912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.5.209.124:45561)
0808 02:12:24.673021 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.673028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.673041 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:12:24.673074 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.673076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.673109 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.673211 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.673655 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.674180 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.675224 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.675567 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.675682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.678104 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.678111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.678117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.678146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.679906 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.680365 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.680370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.680372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.680458 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.680990 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.680992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.680994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.681289 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.681605 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.682291 (+ 686us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.682398 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:12:24.684159 3426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:24.684235 3426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:24.685253 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.673390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.5.209.125:37361)
0808 02:12:24.673516 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.673520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.673562 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:12:24.673722 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.673725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.673765 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.673811 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.674662 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.675444 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.676558 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.676847 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.676957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.680174 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.680178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.680181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.680205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.681983 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.683140 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.683144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.683146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.683237 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.683757 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.683760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.683762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.683969 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.684136 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.684970 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.685105 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:12:24.681679 3222 master_runner.cc:186] Error getting master registration for 127.5.209.126:44611: OK, instance_id { permanent_uuid: "9eba79264d6741418a52921bb603e622" instance_seqno: 1786155144145107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:12:24.685921 3222 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:12:24.686117 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: Bootstrap starting.
I20260808 02:12:24.688318 3401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:24.688880 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:24.689157 3401 log.cc:824] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:24.691522 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: No bootstrap required, opened a new log
I20260808 02:12:24.690429 3180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9eba79264d6741418a52921bb603e622"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43846
I20260808 02:12:24.695765 3401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.696023 3401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e257c215cd34dc3956449d80cda054b, State: Initialized, Role: FOLLOWER
I20260808 02:12:24.696352 3401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.696875 3430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.697027 3430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.697831 3401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:12:24.703614 3443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:24.703701 3443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:24.702008 3315 master_runner.cc:186] Error getting master registration for 127.5.209.126:44611: OK, instance_id { permanent_uuid: "9eba79264d6741418a52921bb603e622" instance_seqno: 1786155144145107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:12:24.705394 3315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:12:24.789891 3214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } attempt: 4
Aug 08 02:12:24 dist-test-slave-bw88 krb5kdc[3108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155144, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
I20260808 02:12:24.799620 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.790530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45561 (local address 127.0.0.1:36678)
0808 02:12:24.790762 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:24.790778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:24.790925 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:24.791241 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:24.791244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:24.791457 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:24.791554 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.791560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.792622 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.792626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.793152 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:24.793160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.793791 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.793795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:24.793879 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.794429 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:24.794442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:24.795932 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:24.797740 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.797745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.797748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.798034 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.798358 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:24.798361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:24.798362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:24.798435 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:24.799009 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:24.799012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:24.799162 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 02:12:24.799288 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 02:12:24.799433 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260808 02:12:24.799623 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.790581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.5.209.124:45561)
0808 02:12:24.790722 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.790727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.790869 (+ 142us) server_negotiation.cc:407] Connection header received
0808 02:12:24.791074 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.791077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.791132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.791211 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.791685 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.792483 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.793294 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.793689 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.793786 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.796090 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.796093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.796095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.796118 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.797620 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.798164 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.798167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.798169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.798254 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.798547 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.798553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.798555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.798871 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.798970 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.799365 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.799481 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:12:24.803870 3214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: Bootstrap starting.
I20260808 02:12:24.806886 3214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:24.807811 3214 log.cc:824] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:24.810318 3214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: No bootstrap required, opened a new log
I20260808 02:12:24.813534 3214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.813792 3214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eba79264d6741418a52921bb603e622, State: Initialized, Role: FOLLOWER
I20260808 02:12:24.814165 3214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.814762 3447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.815012 3447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.815729 3214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:24.815778 3180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:12:24.815917 3180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:24.817329 3180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.818405 3180 leader_election.cc:288] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [CANDIDATE]: Term 1 election: Requested vote from peers f8ad32c83394452b8734e8fde06f338e (127.5.209.125:37361), 5e257c215cd34dc3956449d80cda054b (127.5.209.124:45561)
I20260808 02:12:24.819394 3269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9eba79264d6741418a52921bb603e622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8ad32c83394452b8734e8fde06f338e"
I20260808 02:12:24.819684 3269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:24.819752 3369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9eba79264d6741418a52921bb603e622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e257c215cd34dc3956449d80cda054b"
I20260808 02:12:24.819984 3369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:24.821044 3269 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eba79264d6741418a52921bb603e622 in term 1.
I20260808 02:12:24.821394 3369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eba79264d6741418a52921bb603e622 in term 1.
I20260808 02:12:24.821602 3149 leader_election.cc:302] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eba79264d6741418a52921bb603e622, f8ad32c83394452b8734e8fde06f338e; no voters:
I20260808 02:12:24.821926 3447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:24.822234 3447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [term 1 LEADER]: Becoming Leader. State: Replica: 9eba79264d6741418a52921bb603e622, State: Running, Role: LEADER
I20260808 02:12:24.822629 3447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } }
I20260808 02:12:24.824599 3447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eba79264d6741418a52921bb603e622. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.824761 3447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: This master's current role is: LEADER
W20260808 02:12:24.824970 3464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:24.825088 3464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:24.825286 3472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:24.826308 3472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:24.826783 3127 master_runner.cc:263] No leader master found from master 9eba79264d6741418a52921bb603e622
I20260808 02:12:24.827805 3127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:12:24.832598 3369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [term 1 FOLLOWER]: Refusing update from remote peer 9eba79264d6741418a52921bb603e622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:12:24.833259 3469 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:12:24.834580 3269 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [term 1 FOLLOWER]: Refusing update from remote peer 9eba79264d6741418a52921bb603e622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:12:24.835165 3469 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:12:24.837764 3430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eba79264d6741418a52921bb603e622. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.837906 3430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.839182 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eba79264d6741418a52921bb603e622. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.839318 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.842150 3447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.842264 3447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:24.842489 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.842626 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.842960 3475 mvcc.cc:205] Tried to move back new op lower bound from 7316091473227563008 to 7316091473197604864. Current Snapshot: MvccSnapshot[applied={T|T < 7316091473227563008}]
I20260808 02:12:24.845148 3482 mvcc.cc:205] Tried to move back new op lower bound from 7316091473227563008 to 7316091473197604864. Current Snapshot: MvccSnapshot[applied={T|T < 7316091473227563008}]
I20260808 02:12:24.845781 3447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.845887 3447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:24.846894 3430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eba79264d6741418a52921bb603e622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eba79264d6741418a52921bb603e622" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 44611 } } peers { permanent_uuid: "f8ad32c83394452b8734e8fde06f338e" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 37361 } } peers { permanent_uuid: "5e257c215cd34dc3956449d80cda054b" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45561 } } }
I20260808 02:12:24.847016 3430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:24.851110 3472 catalog_manager.cc:1347] Generated new cluster ID: 6a8b0f6de222457cacf0291eab401036
I20260808 02:12:24.851176 3472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:24.855873 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.829765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.5.209.125:37361)
0808 02:12:24.829882 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.829886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.830929 (+ 1043us) server_negotiation.cc:407] Connection header received
0808 02:12:24.831143 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.831147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.831196 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.831249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.832538 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.833282 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.836937 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.837277 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.837388 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.843081 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.843085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.843088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.843115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.844859 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.849792 (+ 4933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.849798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.849800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.849902 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.854068 (+ 4166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.854072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.854074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.854288 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.854393 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.855590 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.855716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:12:24.859114 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.830191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.5.209.124:45561)
0808 02:12:24.830307 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.830311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.830956 (+ 645us) server_negotiation.cc:407] Connection header received
0808 02:12:24.831146 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.831149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.831196 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.831249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.834971 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.835662 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.838054 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.838371 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.838465 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.847926 (+ 9461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.847930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.847932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.847961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.849655 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.857667 (+ 8012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.857671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.857673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.857772 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.858106 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.858109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.858111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.858340 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.858456 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.858838 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.858965 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 02:12:24.861438 3478 mvcc.cc:205] Tried to move back new op lower bound from 7316091473227563008 to 7316091473197604864. Current Snapshot: MvccSnapshot[applied={T|T < 7316091473227563008 or (T in {7316091473227563008})}]
I20260808 02:12:24.862511 3472 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:24.863860 3472 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:24.867741 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:24.833669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.5.209.126:44611)
0808 02:12:24.833865 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.833870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.842769 (+ 8899us) server_negotiation.cc:407] Connection header received
0808 02:12:24.842950 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.842954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.843014 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.843066 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.850012 (+ 6946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.850761 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.856340 (+ 5579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:24.856669 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:24.856748 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:24.861310 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:24.861314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:24.861316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:24.861381 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:24.864076 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.866491 (+ 2415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.866495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.866496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.866579 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:24.866845 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:24.866848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:24.866849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:24.867026 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:24.867147 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:24.867459 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 02:12:24.867593 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260808 02:12:24.884209 3472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eba79264d6741418a52921bb603e622: Generated new TSK 0
I20260808 02:12:24.885088 3472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:12:24.926908 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.922618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.5.209.125:37361)
0808 02:12:24.922756 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.922761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.925653 (+ 2892us) server_negotiation.cc:407] Connection header received
0808 02:12:24.925863 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.925866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.925921 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.925983 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.926694 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39422: BlockingRecv error: recv got EOF from 127.0.0.1:39422 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 02:12:24.929224 3218 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.924193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.5.209.126:44611)
0808 02:12:24.924306 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.924311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.927780 (+ 3469us) server_negotiation.cc:407] Connection header received
0808 02:12:24.928461 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.928465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.928519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.928569 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.929099 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43858: BlockingRecv error: recv got EOF from 127.0.0.1:43858 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 02:12:24.930025 3406 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.923430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.5.209.124:45561)
0808 02:12:24.923543 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.923547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.927588 (+ 4041us) server_negotiation.cc:407] Connection header received
0808 02:12:24.928061 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.928065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.928117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.928186 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.929899 (+ 1713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36700: BlockingRecv error: recv got EOF from 127.0.0.1:36700 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 02:12:24.934312 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.932741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.5.209.125:37361)
0808 02:12:24.932863 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.932870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.933359 (+ 489us) server_negotiation.cc:407] Connection header received
0808 02:12:24.933575 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.933577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.933644 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.933748 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.934229 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39436: BlockingRecv error: recv got EOF from 127.0.0.1:39436 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 02:12:24.938639 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.937792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.5.209.125:37361)
0808 02:12:24.937917 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.937920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.937931 (+ 11us) server_negotiation.cc:407] Connection header received
0808 02:12:24.937974 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.937976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.938014 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.938114 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.938528 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39450: BlockingRecv error: recv got EOF from 127.0.0.1:39450 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 02:12:24.943712 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.942651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.5.209.125:37361)
0808 02:12:24.942775 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.942780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.942936 (+ 156us) server_negotiation.cc:407] Connection header received
0808 02:12:24.943063 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.943067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.943123 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.943217 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.943605 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39456: BlockingRecv error: recv got EOF from 127.0.0.1:39456 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 02:12:24.950716 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.949485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.5.209.125:37361)
0808 02:12:24.949693 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.949698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.949931 (+ 233us) server_negotiation.cc:407] Connection header received
0808 02:12:24.950045 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.950048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.950103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.950198 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.950605 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39458: BlockingRecv error: recv got EOF from 127.0.0.1:39458 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260808 02:12:24.961704 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.960486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.5.209.125:37361)
0808 02:12:24.960634 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.960639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.960764 (+ 125us) server_negotiation.cc:407] Connection header received
0808 02:12:24.960928 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.960931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.960983 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.961082 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.961583 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39464: BlockingRecv error: recv got EOF from 127.0.0.1:39464 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 02:12:24.981168 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:24.979989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.5.209.125:37361)
0808 02:12:24.980146 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:24.980151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:24.980300 (+ 149us) server_negotiation.cc:407] Connection header received
0808 02:12:24.980406 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:24.980409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:24.980473 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:24.980585 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:24.981029 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39476: BlockingRecv error: recv got EOF from 127.0.0.1:39476 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 02:12:25.016749 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:25.015532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.5.209.125:37361)
0808 02:12:25.015690 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.015695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.015826 (+ 131us) server_negotiation.cc:407] Connection header received
0808 02:12:25.015961 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.015964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.016024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.016128 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:25.016619 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39492: BlockingRecv error: recv got EOF from 127.0.0.1:39492 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 02:12:25.084293 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:25.083132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.5.209.125:37361)
0808 02:12:25.083322 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.083326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.083446 (+ 120us) server_negotiation.cc:407] Connection header received
0808 02:12:25.083579 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.083582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.083646 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.083736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:25.084187 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39506: BlockingRecv error: recv got EOF from 127.0.0.1:39506 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260808 02:12:25.216163 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:25.214875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.5.209.125:37361)
0808 02:12:25.215037 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.215043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.215192 (+ 149us) server_negotiation.cc:407] Connection header received
0808 02:12:25.215341 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.215343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.215396 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.215480 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:25.215971 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39520: BlockingRecv error: recv got EOF from 127.0.0.1:39520 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 02:12:25.475857 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:25.474778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.5.209.125:37361)
0808 02:12:25.474938 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.474942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.474960 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:12:25.475124 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.475126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.475178 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.475266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:25.475754 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39534: BlockingRecv error: recv got EOF from 127.0.0.1:39534 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:12:25.685777 3426 catalog_manager.cc:1259] Loaded cluster ID: 6a8b0f6de222457cacf0291eab401036
I20260808 02:12:25.685873 3426 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: loading cluster ID for follower catalog manager: success
I20260808 02:12:25.687654 3426 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: acquiring CA information for follower catalog manager: success
I20260808 02:12:25.688900 3426 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8ad32c83394452b8734e8fde06f338e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:12:25.704986 3443 catalog_manager.cc:1259] Loaded cluster ID: 6a8b0f6de222457cacf0291eab401036
I20260808 02:12:25.705103 3443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: loading cluster ID for follower catalog manager: success
I20260808 02:12:25.706889 3443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: acquiring CA information for follower catalog manager: success
I20260808 02:12:25.708076 3443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e257c215cd34dc3956449d80cda054b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:12:25.994625 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:25.990315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.5.209.125:37361)
0808 02:12:25.990662 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.990669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.990687 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:12:25.990883 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.990886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.991033 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.991181 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:25.991473 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:25.992024 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:25.992970 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:25.993208 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:25.993331 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:25.993703 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:25.994166 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:25.994313 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:25.994375 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 02:12:25.994445 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260808 02:12:26.001880 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:25.997775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.5.209.124:45561)
0808 02:12:25.998030 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:25.998037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:25.998056 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:12:25.998263 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:25.998266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:25.998389 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:25.998556 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:25.998784 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:25.999393 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.000276 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.000517 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.000633 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:26.000949 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:26.001433 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:26.001625 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:26.001661 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 02:12:26.001714 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260808 02:12:26.009253 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:26.004817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.5.209.125:37361)
0808 02:12:26.004976 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:26.004981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:26.005632 (+ 651us) server_negotiation.cc:407] Connection header received
0808 02:12:26.005945 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:26.005949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:26.006001 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:26.006073 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:26.006474 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.007023 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.007992 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.008266 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.008361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:26.008742 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:26.008828 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:26.008876 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:26.009059 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:12:26.009120 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:12:26.010053 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:26.005700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.5.209.124:45561)
0808 02:12:26.005802 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:26.005806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:26.005953 (+ 147us) server_negotiation.cc:407] Connection header received
0808 02:12:26.006059 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:26.006062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:26.006104 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:26.006191 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:26.006553 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.007242 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.008327 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.008880 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.009066 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:26.009613 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:26.009704 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:26.009773 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:26.009862 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:12:26.009909 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 02:12:26.010458 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:26.004961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.5.209.126:44611)
0808 02:12:26.005213 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:26.005219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:26.005237 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:12:26.005364 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:26.005368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:26.005548 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:26.005964 (+ 416us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:26.006103 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.006932 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.008259 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:26.008619 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:26.009008 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:26.009167 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:26.009653 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:26.009784 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:26.010219 (+ 435us) server_negotiation.cc:299] Negotiation successful
0808 02:12:26.010280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260808 02:12:26.013944 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3127
I20260808 02:12:26.026767 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3222
I20260808 02:12:26.038076 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3315
2026-08-08T02:12:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3462 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:12:26 dist-test-slave-bw88 krb5kdc[3598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:12:26 dist-test-slave-bw88 krb5kdc[3598](info): set up 2 sockets
Aug 08 02:12:26 dist-test-slave-bw88 krb5kdc[3598](info): commencing operation
krb5kdc: starting...
W20260808 02:12:27.239885 5957 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/test-user.keytab.
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.209.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
Entry for principal kudu/127.5.209.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab.
I20260808 02:12:27.381445 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:32787
--webserver_interface=127.5.209.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:12:27.499040 3617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:27.499410 3617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:27.499486 3617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:27.499557 3617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:27.503654 3617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:27.503743 3617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:27.503783 3617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:27.503837 3617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:27.503876 3617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:27.509287 3617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.126.keytab
--principal=kudu/127.5.209.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.126:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.209.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:27.510488 3617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:27.511580 3617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:27.520603 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:27.520789 3617 server_base.cc:1034] running on GCE node
W20260808 02:12:27.520601 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:27.520668 3624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:27.521706 3617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:27.522769 3617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:27.523933 3617 hybrid_clock.cc:648] HybridClock initialized: now 1786155147523914 us; error 31 us; skew 500 ppm
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:27.526868 3617 init.cc:377] Logged in from keytab as kudu/127.5.209.126@KRBTEST.COM (short username kudu)
I20260808 02:12:27.527828 3617 webserver.cc:460] Webserver started at http://127.5.209.126:33621/ using document root <none> and password file <none>
I20260808 02:12:27.528409 3617 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:27.528472 3617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:27.528748 3617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:27.530949 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data/instance:
uuid: "6ae86b430db242c2bd48c47366c5d8e9"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "9488a2444d634fd965f2afbe2dd26fac"
server_key_iv: "f48a9b8616b7bfd6782c6a3bdefa6781"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.531558 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance:
uuid: "6ae86b430db242c2bd48c47366c5d8e9"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "9488a2444d634fd965f2afbe2dd26fac"
server_key_iv: "f48a9b8616b7bfd6782c6a3bdefa6781"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.535429 3617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:12:27.537995 3635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:27.539381 3617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:27.539583 3617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal
uuid: "6ae86b430db242c2bd48c47366c5d8e9"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "9488a2444d634fd965f2afbe2dd26fac"
server_key_iv: "f48a9b8616b7bfd6782c6a3bdefa6781"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.539692 3617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:27.560077 3617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:27.563721 3617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:27.563975 3617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:27.572029 3617 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.126:32787
I20260808 02:12:27.572384 3704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.126:32787 every 8 connection(s)
I20260808 02:12:27.573138 3617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/data/info.pb
I20260808 02:12:27.576262 3706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:27.579504 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3617
I20260808 02:12:27.579612 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-0/wal/instance
I20260808 02:12:27.579923 5957 external_mini_cluster.cc:1428] Setting key bea2886e674965f34fd8859407f84586
I20260808 02:12:27.580725 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.209.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.125@KRBTEST.COM" created.
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:27.599223 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.583414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:32787 (local address 127.0.0.1:33196)
0808 02:12:27.583999 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.584069 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:27.585013 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:27.585872 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:27.585878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:27.586381 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:27.586641 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.586656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.587967 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.587976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.588781 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.588791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.589699 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.589704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.589834 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.590512 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:27.590530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:27.593477 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:27.595816 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.595828 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.595839 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.596111 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.596498 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.596503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.596505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.596637 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.597403 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:27.597410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:27.597572 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 02:12:27.597774 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 02:12:27.598282 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":91,"thread_start_us":154,"threads_started":1}
I20260808 02:12:27.599226 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.583414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.5.209.126:32787)
0808 02:12:27.584010 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:27.584030 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:27.584204 (+ 174us) server_negotiation.cc:407] Connection header received
0808 02:12:27.585430 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:27.585436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:27.585656 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:27.585876 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:27.586818 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.587801 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.588923 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.589505 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.589713 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.593681 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:27.593687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:27.593699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:27.593722 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:27.595659 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.596245 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.596251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.596253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.596378 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.596750 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.596756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.596758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.597197 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:27.597416 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:27.597847 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 02:12:27.598401 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":492,"spinlock_wait_cycles":24320,"thread_start_us":220,"threads_started":1}
I20260808 02:12:27.602221 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:27.603029 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.602757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52118)
0808 02:12:27.602893 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.602968 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 02:12:27.603358 3639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
I20260808 02:12:27.604434 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.604149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52126)
0808 02:12:27.604304 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.604372 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 02:12:27.604672 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:42829: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
Entry for principal kudu/127.5.209.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab.
I20260808 02:12:27.613302 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:42829
--webserver_interface=127.5.209.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:12:27.653959 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } attempt: 1
I20260808 02:12:27.654953 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.654596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52138)
0808 02:12:27.654801 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.654869 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:12:27.655784 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.655569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52146)
0808 02:12:27.655684 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.655730 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:12:27.655952 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:42829: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
I20260808 02:12:27.724112 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } attempt: 2
I20260808 02:12:27.725145 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.724733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52156)
0808 02:12:27.724966 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.725027 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260808 02:12:27.725863 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.725653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52168)
0808 02:12:27.725759 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.725807 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 02:12:27.726058 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:42829: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
W20260808 02:12:27.740836 3716 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:27.741262 3716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:27.741364 3716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:27.741439 3716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:27.748533 3716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:27.748621 3716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:27.748663 3716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:27.748695 3716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:27.748735 3716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:27.758939 3716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.125.keytab
--principal=kudu/127.5.209.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.125:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.209.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:27.760568 3716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:27.762148 3716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:27.773934 3725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:27.773916 3724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:27.774111 3727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:27.775214 3716 server_base.cc:1034] running on GCE node
I20260808 02:12:27.775965 3716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:27.777549 3716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:27.778791 3716 hybrid_clock.cc:648] HybridClock initialized: now 1786155147778751 us; error 53 us; skew 500 ppm
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:27.783258 3716 init.cc:377] Logged in from keytab as kudu/127.5.209.125@KRBTEST.COM (short username kudu)
I20260808 02:12:27.784428 3716 webserver.cc:460] Webserver started at http://127.5.209.125:40793/ using document root <none> and password file <none>
I20260808 02:12:27.785248 3716 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:27.785383 3716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:27.785651 3716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:27.788475 3716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data/instance:
uuid: "5be98bdd9574433ba6414af2c7db577f"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "412c94ce09b2f99ef777b4dde735ab9e"
server_key_iv: "c0a69afb4340131585f72d091edd0207"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.789271 3716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal/instance:
uuid: "5be98bdd9574433ba6414af2c7db577f"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "412c94ce09b2f99ef777b4dde735ab9e"
server_key_iv: "c0a69afb4340131585f72d091edd0207"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.794123 3716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:12:27.798115 3735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:27.799852 3716 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 02:12:27.800015 3716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal
uuid: "5be98bdd9574433ba6414af2c7db577f"
format_stamp: "Formatted at 2026-08-08 02:12:27 on dist-test-slave-bw88"
server_key: "412c94ce09b2f99ef777b4dde735ab9e"
server_key_iv: "c0a69afb4340131585f72d091edd0207"
server_key_version: "encryptionkey@0"
I20260808 02:12:27.800158 3716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:27.817745 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } attempt: 3
I20260808 02:12:27.818578 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.818240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52180)
0808 02:12:27.818441 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.818493 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260808 02:12:27.819422 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.819145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52182)
0808 02:12:27.819301 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.819368 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 02:12:27.819581 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.125:42829: Network error: Client connection negotiation failed: client connection to 127.5.209.125:42829: connect: Connection refused (error 111)
I20260808 02:12:27.826450 3716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:27.829545 3716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:27.829753 3716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:27.837196 3716 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.125:42829
I20260808 02:12:27.837584 3800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.125:42829 every 8 connection(s)
I20260808 02:12:27.838304 3716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/data/info.pb
I20260808 02:12:27.841207 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:27.841223 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3716
I20260808 02:12:27.841394 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-1/wal/instance
I20260808 02:12:27.841693 5957 external_mini_cluster.cc:1428] Setting key 6b06bee42398d3b4dd5d9ef7cd1f81b4
I20260808 02:12:27.845623 3802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.209.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.209.124@KRBTEST.COM" created.
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:27.860850 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.847737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.5.209.126:32787)
0808 02:12:27.847883 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:27.847887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:27.848995 (+ 1108us) server_negotiation.cc:407] Connection header received
0808 02:12:27.850073 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:27.850077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:27.850151 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:27.850260 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:27.851496 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.852043 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.853291 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.853642 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.853749 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.856654 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:27.856657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:27.856659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:27.856685 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:27.858436 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.858966 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.858973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.858977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.859088 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.859441 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.859447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.859451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.859749 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:27.859876 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:27.860503 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 02:12:27.860655 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:12:27.861989 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.848253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:32787 (local address 127.0.0.1:33198)
0808 02:12:27.848826 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.848872 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:27.849832 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:27.850447 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:27.850454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:27.850805 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:27.851302 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.851315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.852189 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.852193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.853148 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.853158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.853849 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.853857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.854026 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.854723 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:27.854750 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:27.856431 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:27.858559 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.858565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.858578 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.858849 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.859192 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.859195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.859196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.859335 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.859876 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:27.859881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:27.860215 (+ 334us) client_negotiation.cc:769] Sending connection context
0808 02:12:27.860386 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 02:12:27.860662 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":171,"threads_started":1}
I20260808 02:12:27.863659 3802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:27.873842 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.864267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52196)
0808 02:12:27.864397 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.864413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:27.864499 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:27.865107 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:27.865110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:27.865247 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:27.865426 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.865433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.866556 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.866560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.867358 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.867372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.867986 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.867989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.868147 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.868587 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:27.868599 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:27.870003 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:27.872165 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.872168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.872169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.872366 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.872704 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.872707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.872709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.872813 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.873278 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:27.873281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:27.873422 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:12:27.873521 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:12:27.873648 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
Entry for principal kudu/127.5.209.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab.
I20260808 02:12:27.874058 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.864359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.5.209.125:42829)
0808 02:12:27.864752 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:27.864761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:27.864779 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:12:27.864812 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:27.864816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:27.864963 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:27.865069 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:27.865553 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.866420 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.867493 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.867870 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.867954 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.870189 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:27.870199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:27.870212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:27.870256 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:27.872061 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.872490 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.872495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.872496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.872589 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.872906 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.872908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.872909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.873166 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:27.873323 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:27.873633 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 02:12:27.873792 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":147,"threads_started":1}
I20260808 02:12:27.875174 5957 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskguwdHL/build/debug/bin/kudu
/tmp/dist-test-taskguwdHL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:45543
--webserver_interface=127.5.209.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.209.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:12:27.876250 3802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:27.877115 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.876745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53782)
0808 02:12:27.876906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.877013 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 02:12:27.877327 3738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
I20260808 02:12:27.878341 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.878113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53798)
0808 02:12:27.878233 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.878282 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:12:27.878520 3802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45543: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
I20260808 02:12:27.927767 3802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } attempt: 1
I20260808 02:12:27.928797 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.928468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53800)
0808 02:12:27.928638 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.928709 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 02:12:27.929701 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.929436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53814)
0808 02:12:27.929594 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.929644 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 02:12:27.929847 3802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45543: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
I20260808 02:12:27.962759 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } attempt: 4
Aug 08 02:12:27 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:27.973239 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.963588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.5.209.125:42829)
0808 02:12:27.963762 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:27.963766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:27.963847 (+ 81us) server_negotiation.cc:407] Connection header received
0808 02:12:27.964011 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:27.964013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:27.964066 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:27.964142 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:27.964782 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.965316 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.966504 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.966744 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.966827 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.969411 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:27.969415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:27.969417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:27.969441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:27.971461 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.971939 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.971944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.971946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.972038 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:27.972354 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:27.972357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:27.972359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:27.972552 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:27.972667 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:27.972965 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:12:27.973098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:12:27.973239 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.963432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52200)
0808 02:12:27.963734 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.963755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:27.963899 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:27.964202 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:27.964207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:27.964461 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:27.964635 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.964645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.965505 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.965509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.966354 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:27.966368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:27.966889 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:27.966893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:27.967022 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:27.967553 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:27.967569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:27.969196 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:27.971589 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.971593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.971595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.971826 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.972151 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:27.972155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:27.972157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:27.972251 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:27.972670 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:27.972675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:27.972790 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 02:12:27.972909 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:12:27.973057 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260808 02:12:27.974246 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:27.975018 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.974752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53828)
0808 02:12:27.974888 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.974945 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:12:27.975855 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.975640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53836)
0808 02:12:27.975758 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.975803 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:12:27.976087 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45543: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
I20260808 02:12:27.998023 3802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } attempt: 2
I20260808 02:12:27.998920 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.998602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53848)
0808 02:12:27.998764 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.998835 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 02:12:27.999800 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:27.999566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53854)
0808 02:12:27.999690 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:27.999745 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:12:27.999974 3802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45543: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
W20260808 02:12:28.002043 3810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:12:28.002455 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:12:28.002524 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:12:28.002599 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:12:28.006906 3810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:12:28.006999 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:12:28.007056 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:12:28.007099 3810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:12:28.007154 3810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:12:28.012912 3810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.209.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.209.126:32787,127.5.209.125:42829,127.5.209.124:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/krb5kdc/kudu_127.5.209.124.keytab
--principal=kudu/127.5.209.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.209.124:45543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.209.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 01:19:23 UTC on ccycloud.quasar-jszeeg.root.comops.site
build id 4573756
I20260808 02:12:28.014137 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:12:28.015270 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:12:28.024613 3824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:28.024730 3810 server_base.cc:1034] running on GCE node
W20260808 02:12:28.024601 3822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:12:28.024601 3819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:12:28.025624 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:12:28.026728 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:12:28.027907 3810 hybrid_clock.cc:648] HybridClock initialized: now 1786155148027896 us; error 27 us; skew 500 ppm
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:12:28.031013 3810 init.cc:377] Logged in from keytab as kudu/127.5.209.124@KRBTEST.COM (short username kudu)
I20260808 02:12:28.032006 3810 webserver.cc:460] Webserver started at http://127.5.209.124:42201/ using document root <none> and password file <none>
I20260808 02:12:28.032629 3810 fs_manager.cc:359] Metadata directory not provided
I20260808 02:12:28.032697 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:12:28.032974 3810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:12:28.035298 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } attempt: 1
I20260808 02:12:28.035537 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data/instance:
uuid: "4d5322261cef4fad9cf13f5eb9679ab0"
format_stamp: "Formatted at 2026-08-08 02:12:28 on dist-test-slave-bw88"
server_key: "141facd7d71e307cb8e026514aaf8931"
server_key_iv: "2f7ba4d707400ae1e1ab65ceb4c1c4ee"
server_key_version: "encryptionkey@0"
I20260808 02:12:28.036237 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.035833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53862)
0808 02:12:28.036053 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.036134 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 02:12:28.036302 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal/instance:
uuid: "4d5322261cef4fad9cf13f5eb9679ab0"
format_stamp: "Formatted at 2026-08-08 02:12:28 on dist-test-slave-bw88"
server_key: "141facd7d71e307cb8e026514aaf8931"
server_key_iv: "2f7ba4d707400ae1e1ab65ceb4c1c4ee"
server_key_version: "encryptionkey@0"
I20260808 02:12:28.037171 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.036896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53878)
0808 02:12:28.037033 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.037093 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:12:28.037317 3706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.209.124:45543: Network error: Client connection negotiation failed: client connection to 127.5.209.124:45543: connect: Connection refused (error 111)
I20260808 02:12:28.040952 3810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:12:28.043931 3832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:12:28.045588 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:12:28.045720 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data,/tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal
uuid: "4d5322261cef4fad9cf13f5eb9679ab0"
format_stamp: "Formatted at 2026-08-08 02:12:28 on dist-test-slave-bw88"
server_key: "141facd7d71e307cb8e026514aaf8931"
server_key_iv: "2f7ba4d707400ae1e1ab65ceb4c1c4ee"
server_key_version: "encryptionkey@0"
I20260808 02:12:28.045819 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:12:28.060437 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:12:28.063971 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:12:28.064142 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:12:28.071816 3810 rpc_server.cc:307] RPC server started. Bound to: 127.5.209.124:45543
I20260808 02:12:28.072114 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.209.124:45543 every 8 connection(s)
I20260808 02:12:28.072889 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/data/info.pb
I20260808 02:12:28.073683 5957 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskguwdHL/build/debug/bin/kudu as pid 3810
I20260808 02:12:28.073845 5957 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskguwdHL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786154850122309-5957-0/minicluster-data/master-2/wal/instance
I20260808 02:12:28.074153 5957 external_mini_cluster.cc:1428] Setting key 3e3586fdfd341a5692ca0c7b6085a31b
I20260808 02:12:28.076282 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:12:28.080827 3908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
I20260808 02:12:28.090355 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.075541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.5.209.124:45543)
0808 02:12:28.075969 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.075986 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.076024 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:12:28.076758 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.076777 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.077128 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.077494 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.078446 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.079477 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.080657 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.081232 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.081446 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.084358 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.084363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.084408 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.084453 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.086963 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.087458 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.087463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.087468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.087588 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.087929 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.087932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.087934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.088551 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.088769 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.089093 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.089487 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":142,"threads_started":1}
I20260808 02:12:28.091147 3802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } attempt: 3
I20260808 02:12:28.094517 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.083092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.126:32787 (local address 127.0.0.1:33206)
0808 02:12:28.083990 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.084033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:28.084145 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:28.084461 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:28.084467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:28.085053 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:28.085255 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.085265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.086070 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.086075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.087186 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.087195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.087717 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.087720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.087813 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.088340 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:28.088355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:28.089973 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:28.092581 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.092589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.092594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.092847 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.093218 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.093221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.093223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.093354 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.093790 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:28.093797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:28.093924 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 02:12:28.094093 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 02:12:28.094296 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":120,"threads_started":1}
I20260808 02:12:28.094517 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.083175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.5.209.126:32787)
0808 02:12:28.083338 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.083343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.084131 (+ 788us) server_negotiation.cc:407] Connection header received
0808 02:12:28.084268 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.084271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.084330 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.084394 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.085439 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.085925 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.087306 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.087591 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.087711 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.090134 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.090144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.090147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.090171 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.092438 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.092979 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.092985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.092987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.093090 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.093446 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.093450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.093451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.093666 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.093790 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.094206 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.094327 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:12:28.095760 3908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.125@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.126@KRBTEST.COM
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:28.103444 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.091701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.5.209.124:45543)
0808 02:12:28.091810 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.091814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.092621 (+ 807us) server_negotiation.cc:407] Connection header received
0808 02:12:28.092664 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.092666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.092712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.093375 (+ 663us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.093519 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.094331 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.095562 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.095932 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.096046 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.098821 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.098825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.098828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.098855 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.100795 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.101736 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.101740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.101742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.101834 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.102190 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.102224 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.102226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.102806 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.102899 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.103190 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.103285 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:12:28.103463 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.091701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53900)
0808 02:12:28.092399 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.092417 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:28.092527 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:28.092856 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:28.092860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:28.093097 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:28.093235 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.093243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.094561 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.094565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.095388 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.095400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.096069 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.096073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.096160 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.096629 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:28.096646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:28.098640 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:28.101233 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.101237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.101239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.101596 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.101962 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.101966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.101967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.102081 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.102901 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:28.102906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:28.103001 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 02:12:28.103133 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 02:12:28.103272 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588}
I20260808 02:12:28.104434 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.093328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.5.209.126:32787)
0808 02:12:28.093671 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.093684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.093698 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:12:28.093736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.093739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.093784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.093877 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.094630 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.095388 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.096313 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.096695 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.096818 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.100026 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.100032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.100036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.100070 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.102161 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.102652 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.102659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.102664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.102768 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.103114 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.103119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.103124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.103338 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.104023 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.104112 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.104237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260808 02:12:28.104498 3706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } attempt: 2
I20260808 02:12:28.106343 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.096529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.125:42829 (local address 127.0.0.1:52210)
0808 02:12:28.096637 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.096653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:28.096726 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:28.097051 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:28.097054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:28.097216 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:28.097390 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.097402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.098142 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.098149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.098734 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.098773 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.099419 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.099424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.099560 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.100215 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:28.100232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:28.101577 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:28.104493 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.104496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.104498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.104781 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.105176 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.105180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.105181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.105279 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.105871 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:28.105874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:28.105965 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 02:12:28.106058 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:12:28.106196 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 02:12:28.107056 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.096727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.5.209.125:42829)
0808 02:12:28.096845 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.096849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.096869 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:12:28.096896 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.096898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.096929 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.097026 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.097534 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.098010 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.098918 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.099305 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.099421 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.101722 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.101725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.101728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.101751 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.104356 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.104915 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.104921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.104923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.105052 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.105418 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.105422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.105426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.105731 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.105824 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.106775 (+ 951us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.106889 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:12:28.107999 3908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:12:28.108474 3802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: Bootstrap starting.
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.126@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
I20260808 02:12:28.112449 3802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:28.113525 3802 log.cc:824] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155148, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.209.124@KRBTEST.COM for kudu/127.5.209.124@KRBTEST.COM
Aug 08 02:12:28 dist-test-slave-bw88 krb5kdc[3598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.209.125@KRBTEST.COM
I20260808 02:12:28.116102 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.105061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53910)
0808 02:12:28.105281 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.105297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:28.105399 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:28.105844 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:28.105847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:28.106080 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:28.106195 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.106204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.107224 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.107229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.108094 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.108106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.109483 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.109487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.109583 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.110112 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:28.110128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:28.111644 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:28.114145 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.114150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.114154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.114455 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.114878 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.114885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.114889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.114995 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.115581 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:28.115584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:28.115688 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 02:12:28.115806 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:12:28.115946 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260808 02:12:28.116189 3802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: No bootstrap required, opened a new log
I20260808 02:12:28.116217 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.105146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.5.209.124:45543)
0808 02:12:28.105610 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.105615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.105628 (+ 13us) server_negotiation.cc:407] Connection header received
0808 02:12:28.105662 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.105665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.105706 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.106350 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.106467 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.107096 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.108947 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.109303 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.110728 (+ 1425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.111805 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.111809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.111812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.111839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.113984 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.114611 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.114618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.114623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.114746 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.115138 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.115147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.115151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.115385 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.115564 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.115909 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.116035 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260808 02:12:28.118953 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.108464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.209.124:45543 (local address 127.0.0.1:53920)
0808 02:12:28.108620 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:12:28.108635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:12:28.108706 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:12:28.109434 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:12:28.109436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:12:28.109601 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:12:28.109681 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.109688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.110755 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.110762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.111567 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:12:28.111579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.112234 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.112237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:12:28.112343 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.112908 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:12:28.112922 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:12:28.114423 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:12:28.117577 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.117580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.117582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.117798 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.118058 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:12:28.118061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:12:28.118063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:12:28.118134 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:12:28.118498 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:12:28.118501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:12:28.118605 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 02:12:28.118684 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:12:28.118802 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:12:28.119920 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.108690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.5.209.124:45543)
0808 02:12:28.109170 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.109181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.109195 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:12:28.109227 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.109230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.109268 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.109780 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.109865 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.110635 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.111766 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.112115 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.112616 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.114573 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.114580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.114582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.114603 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.117459 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.117899 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.117902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.117904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.117975 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.118222 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.118225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.118227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.118398 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.119485 (+ 1087us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.119580 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.119715 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":134,"threads_started":1}
I20260808 02:12:28.120867 3802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.121109 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.106261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.5.209.125:42829)
0808 02:12:28.107663 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.107667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.107679 (+ 12us) server_negotiation.cc:407] Connection header received
0808 02:12:28.107709 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.107712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.107859 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.107928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.109267 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.110205 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.111283 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.111645 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.111759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.116420 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.116424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.116427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.116454 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.118760 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.119468 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.119472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.119473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.119567 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.119912 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.119915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.119917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.120164 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.120322 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.120698 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.120819 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":145,"threads_started":1}
I20260808 02:12:28.121246 3706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: Bootstrap starting.
I20260808 02:12:28.121328 3802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be98bdd9574433ba6414af2c7db577f, State: Initialized, Role: FOLLOWER
I20260808 02:12:28.121770 3802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.122442 3922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.122663 3922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.123296 3802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:28.124468 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: Bootstrap starting.
I20260808 02:12:28.125161 5957 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:12:28.127218 3908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:28.126518 3668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ae86b430db242c2bd48c47366c5d8e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33220
I20260808 02:12:28.128126 3908 log.cc:824] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: Log is configured to *not* fsync() on all Append() calls
W20260808 02:12:28.129009 3934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:28.129179 3934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:28.129411 3716 master_runner.cc:186] Error getting master registration for 127.5.209.126:32787: OK, instance_id { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" instance_seqno: 1786155147569975 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:12:28.131865 3716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:12:28.132347 3706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: Neither blocks nor log segments found. Creating new log.
I20260808 02:12:28.133078 3706 log.cc:824] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:12:28.135869 3706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: No bootstrap required, opened a new log
I20260808 02:12:28.135869 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: No bootstrap required, opened a new log
I20260808 02:12:28.139057 3706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.139302 3706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae86b430db242c2bd48c47366c5d8e9, State: Initialized, Role: FOLLOWER
I20260808 02:12:28.139664 3706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.139976 3908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.140226 3908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5322261cef4fad9cf13f5eb9679ab0, State: Initialized, Role: FOLLOWER
I20260808 02:12:28.140440 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.140533 3908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.140789 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.141139 3706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:28.141247 3668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:12:28.141477 3668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:28.141849 3908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:12:28.142892 3668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.143882 3668 leader_election.cc:288] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [CANDIDATE]: Term 1 election: Requested vote from peers 5be98bdd9574433ba6414af2c7db577f (127.5.209.125:42829), 4d5322261cef4fad9cf13f5eb9679ab0 (127.5.209.124:45543)
I20260808 02:12:28.145264 3768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ae86b430db242c2bd48c47366c5d8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5be98bdd9574433ba6414af2c7db577f"
I20260808 02:12:28.145547 3768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:28.145820 3941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.145942 3941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.146971 3768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae86b430db242c2bd48c47366c5d8e9 in term 1.
I20260808 02:12:28.147439 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ae86b430db242c2bd48c47366c5d8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d5322261cef4fad9cf13f5eb9679ab0"
I20260808 02:12:28.147647 3870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:12:28.148942 3870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae86b430db242c2bd48c47366c5d8e9 in term 1.
I20260808 02:12:28.150439 3637 leader_election.cc:302] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d5322261cef4fad9cf13f5eb9679ab0, 6ae86b430db242c2bd48c47366c5d8e9; no voters:
W20260808 02:12:28.152029 3971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:28.152092 3971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:28.153712 3810 master_runner.cc:263] No leader master found from master 4d5322261cef4fad9cf13f5eb9679ab0
I20260808 02:12:28.153779 3810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:12:28.153928 3940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:12:28.154707 3940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6ae86b430db242c2bd48c47366c5d8e9, State: Running, Role: LEADER
I20260808 02:12:28.155148 3940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } }
I20260808 02:12:28.155874 3617 master_runner.cc:263] No leader master found from master 6ae86b430db242c2bd48c47366c5d8e9
I20260808 02:12:28.155941 3617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 02:12:28.160663 3970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:12:28.160768 3970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:12:28.161108 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ae86b430db242c2bd48c47366c5d8e9. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.161280 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:28.164144 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.152438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.5.209.126:32787)
0808 02:12:28.152551 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.152555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.153179 (+ 624us) server_negotiation.cc:407] Connection header received
0808 02:12:28.153421 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.153427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.153477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.153545 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.154869 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.155640 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.157871 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.158215 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.158297 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.160235 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.160239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.160241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.160267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.162131 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.162660 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.162664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.162666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.162759 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.163094 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.163098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.163100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.163325 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.163442 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.163855 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.163987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 02:12:28.168579 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.157733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.5.209.124:45543)
0808 02:12:28.157852 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.157857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.158481 (+ 624us) server_negotiation.cc:407] Connection header received
0808 02:12:28.158672 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.158676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.158723 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.158779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.159836 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.160596 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.162007 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.162376 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.162474 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.164518 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.164523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.164526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.164556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.166468 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.167020 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.167024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.167026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.167124 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.167476 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.167480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.167482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.167727 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.167856 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.168245 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.168381 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:12:28.170743 3988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:12:28.172114 3988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:12:28.175961 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:28.154328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.5.209.125:42829)
0808 02:12:28.154470 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.154475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.154998 (+ 523us) server_negotiation.cc:407] Connection header received
0808 02:12:28.155196 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.155199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.155251 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.155314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.161605 (+ 6291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.162352 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.169455 (+ 7103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:28.169783 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:28.169879 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:28.171921 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:12:28.171925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:12:28.171928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:12:28.171961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:12:28.173744 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.174298 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.174301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.174303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.174417 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:12:28.174812 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:12:28.174816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:12:28.174818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:12:28.175068 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:12:28.175209 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:28.175645 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 02:12:28.175782 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:12:28.178949 3870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [term 1 FOLLOWER]: Refusing update from remote peer 6ae86b430db242c2bd48c47366c5d8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:12:28.179173 3768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [term 1 FOLLOWER]: Refusing update from remote peer 6ae86b430db242c2bd48c47366c5d8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:12:28.179826 3981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:12:28.180011 3940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:12:28.184327 3922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ae86b430db242c2bd48c47366c5d8e9. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.184494 3922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.184723 3941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ae86b430db242c2bd48c47366c5d8e9. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.184888 3941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.187686 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.187824 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:28.188498 3990 mvcc.cc:205] Tried to move back new op lower bound from 7316091486931128320 to 7316091486848147456. Current Snapshot: MvccSnapshot[applied={T|T < 7316091486931128320}]
I20260808 02:12:28.188957 3922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.189090 3922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.194732 3992 mvcc.cc:205] Tried to move back new op lower bound from 7316091486931128320 to 7316091486848147456. Current Snapshot: MvccSnapshot[applied={T|T < 7316091486931128320}]
I20260808 02:12:28.195601 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.195746 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:12:28.197111 3941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae86b430db242c2bd48c47366c5d8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae86b430db242c2bd48c47366c5d8e9" member_type: VOTER last_known_addr { host: "127.5.209.126" port: 32787 } } peers { permanent_uuid: "5be98bdd9574433ba6414af2c7db577f" member_type: VOTER last_known_addr { host: "127.5.209.125" port: 42829 } } peers { permanent_uuid: "4d5322261cef4fad9cf13f5eb9679ab0" member_type: VOTER last_known_addr { host: "127.5.209.124" port: 45543 } } }
I20260808 02:12:28.197252 3941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:12:28.199697 3991 mvcc.cc:205] Tried to move back new op lower bound from 7316091486931128320 to 7316091486848147456. Current Snapshot: MvccSnapshot[applied={T|T < 7316091486931128320}]
I20260808 02:12:28.200273 3988 catalog_manager.cc:1347] Generated new cluster ID: 800b8f26a359441ca34ad22285b9553c
I20260808 02:12:28.200350 3988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:12:28.240774 3988 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:12:28.242292 3988 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:12:28.263378 3988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ae86b430db242c2bd48c47366c5d8e9: Generated new TSK 0
I20260808 02:12:28.264358 3988 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:12:28.300117 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.296649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.5.209.125:42829)
0808 02:12:28.296788 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.296794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.298915 (+ 2121us) server_negotiation.cc:407] Connection header received
0808 02:12:28.299152 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.299156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.299211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.299272 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.299962 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52240: BlockingRecv error: recv got EOF from 127.0.0.1:52240 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 02:12:28.301715 3915 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.297587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.5.209.126:32787)
0808 02:12:28.297731 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.297735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.300659 (+ 2924us) server_negotiation.cc:407] Connection header received
0808 02:12:28.300850 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.300854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.300921 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.300979 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.301596 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33236: BlockingRecv error: recv got EOF from 127.0.0.1:33236 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 02:12:28.303293 3917 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.298297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.5.209.124:45543)
0808 02:12:28.298437 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.298444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.302314 (+ 3870us) server_negotiation.cc:407] Connection header received
0808 02:12:28.302507 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.302511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.302567 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.302625 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.303165 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53942: BlockingRecv error: recv got EOF from 127.0.0.1:53942 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 02:12:28.308010 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.306630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.5.209.125:42829)
0808 02:12:28.306755 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.306760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.307046 (+ 286us) server_negotiation.cc:407] Connection header received
0808 02:12:28.307269 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.307272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.307348 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.307406 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.307907 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52248: BlockingRecv error: recv got EOF from 127.0.0.1:52248 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 02:12:28.313297 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.311701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.5.209.125:42829)
0808 02:12:28.311806 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.311810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.312353 (+ 543us) server_negotiation.cc:407] Connection header received
0808 02:12:28.312560 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.312564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.312616 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.312671 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.313197 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52256: BlockingRecv error: recv got EOF from 127.0.0.1:52256 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 02:12:28.319550 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.317888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.5.209.125:42829)
0808 02:12:28.318011 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.318016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.318568 (+ 552us) server_negotiation.cc:407] Connection header received
0808 02:12:28.318779 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.318782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.318838 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.318906 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.319446 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52268: BlockingRecv error: recv got EOF from 127.0.0.1:52268 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 02:12:28.331254 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.329294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.5.209.125:42829)
0808 02:12:28.329494 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.329499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.330164 (+ 665us) server_negotiation.cc:407] Connection header received
0808 02:12:28.330396 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.330400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.330464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.330524 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.331129 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52278: BlockingRecv error: recv got EOF from 127.0.0.1:52278 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260808 02:12:28.344728 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.342209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.5.209.125:42829)
0808 02:12:28.342351 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.342357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.343237 (+ 880us) server_negotiation.cc:407] Connection header received
0808 02:12:28.343608 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.343613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.343684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.343754 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.344597 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52288: BlockingRecv error: recv got EOF from 127.0.0.1:52288 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 02:12:28.365561 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.363501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.5.209.125:42829)
0808 02:12:28.363647 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.363652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.364377 (+ 725us) server_negotiation.cc:407] Connection header received
0808 02:12:28.364619 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.364623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.364685 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.364746 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.365423 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52304: BlockingRecv error: recv got EOF from 127.0.0.1:52304 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 02:12:28.401768 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.400549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.5.209.125:42829)
0808 02:12:28.400716 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.400720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.400843 (+ 123us) server_negotiation.cc:407] Connection header received
0808 02:12:28.401000 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.401002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.401057 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.401161 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.401665 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52320: BlockingRecv error: recv got EOF from 127.0.0.1:52320 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 02:12:28.469327 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.468146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.5.209.125:42829)
0808 02:12:28.468325 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.468329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.468432 (+ 103us) server_negotiation.cc:407] Connection header received
0808 02:12:28.468570 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.468573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.468638 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.468728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.469218 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52322: BlockingRecv error: recv got EOF from 127.0.0.1:52322 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 02:12:28.600965 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.599782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.5.209.125:42829)
0808 02:12:28.599975 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.599979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.600088 (+ 109us) server_negotiation.cc:407] Connection header received
0808 02:12:28.600211 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.600213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.600267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.600355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.600858 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52332: BlockingRecv error: recv got EOF from 127.0.0.1:52332 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260808 02:12:28.860483 3807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:12:28.859351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.5.209.125:42829)
0808 02:12:28.859531 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:28.859536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:28.859553 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:12:28.859687 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:28.859690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:28.859743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:28.859816 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:12:28.860350 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52348: BlockingRecv error: recv got EOF from 127.0.0.1:52348 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 02:12:29.130730 3934 catalog_manager.cc:1259] Loaded cluster ID: 800b8f26a359441ca34ad22285b9553c
I20260808 02:12:29.130818 3934 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: loading cluster ID for follower catalog manager: success
I20260808 02:12:29.132556 3934 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: acquiring CA information for follower catalog manager: success
I20260808 02:12:29.133903 3934 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5be98bdd9574433ba6414af2c7db577f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:12:29.153622 3971 catalog_manager.cc:1259] Loaded cluster ID: 800b8f26a359441ca34ad22285b9553c
I20260808 02:12:29.153712 3971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: loading cluster ID for follower catalog manager: success
I20260808 02:12:29.155390 3971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: acquiring CA information for follower catalog manager: success
I20260808 02:12:29.156584 3971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d5322261cef4fad9cf13f5eb9679ab0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:12:29.379074 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:29.374826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.5.209.125:42829)
0808 02:12:29.375104 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:29.375109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:29.375196 (+ 87us) server_negotiation.cc:407] Connection header received
0808 02:12:29.375386 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:29.375390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:29.375532 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:29.375669 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:29.375950 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.376497 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.377466 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.377706 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.377828 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:29.378182 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:29.378645 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:29.378798 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:29.378843 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 02:12:29.378901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260808 02:12:29.386471 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:29.382692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.5.209.124:45543)
0808 02:12:29.382947 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:29.382950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:29.382964 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:12:29.383066 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:29.383068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:29.383188 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:29.383295 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:29.383531 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.384041 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.384915 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.385149 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.385235 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:29.385645 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:29.386103 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:29.386205 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:29.386295 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:12:29.386345 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260808 02:12:29.394766 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:29.390113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.5.209.125:42829)
0808 02:12:29.390262 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:29.390269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:29.390777 (+ 508us) server_negotiation.cc:407] Connection header received
0808 02:12:29.390904 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:29.390907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:29.390966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:29.391061 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:29.391332 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.391989 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.393425 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.393755 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.393856 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:29.394198 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:29.394293 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:29.394375 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:29.394540 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:12:29.394612 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:12:29.395009 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:29.390117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.5.209.124:45543)
0808 02:12:29.390233 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:29.390238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:29.390804 (+ 566us) server_negotiation.cc:407] Connection header received
0808 02:12:29.390984 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:29.390987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:29.391062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:29.391131 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:29.391475 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.392263 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.393620 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.393920 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.394044 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:29.394305 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:29.394394 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:29.394466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:29.394787 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 02:12:29.394851 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 02:12:29.395051 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:12:29.390365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.5.209.126:32787)
0808 02:12:29.390658 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 02:12:29.390663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:12:29.390683 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:12:29.390807 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:12:29.390813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:12:29.391002 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:12:29.391136 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:12:29.391539 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.392339 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.393188 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:12:29.393539 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:12:29.393634 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:12:29.394052 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:12:29.394450 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:12:29.394552 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:12:29.394620 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:12:29.394676 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260808 02:12:29.398563 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3617
I20260808 02:12:29.415063 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3716
I20260808 02:12:29.427886 5957 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskguwdHL/build/debug/bin/kudu with pid 3810
2026-08-08T02:12:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3392 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6855 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299302 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST